Rotor rotation speed driving harmonic wave suppression based audio-visual fusion unmanned aerial vehicle target tracking method and system

Abstract

The application discloses an audio-visual fusion unmanned aerial vehicle target tracking method and system based on rotor speed driving harmonic suppression, and the method comprises the following steps: acquiring multi-modal data; generating a harmonic suppression mask based on rotor speed data; generating a signal-to-noise ratio adaptive weight based on multi-channel microphone data, and jointly generating a weighted cross-correlation spectrum with a frequency domain mask to estimate acoustic uncertainty; performing target detection on target image data to determine a visual direction, and outputting visual uncertainty from an evidence regression model; determining the corresponding observation noise covariance according to the acoustic uncertainty and the visual uncertainty, and inputting the effective observation into a state estimation filter to perform sequential measurement update through a gating judgment; calculating fusion confidence based on fusion covariance, and adaptively scaling the control gain to generate a tracking control instruction according to the fusion direction and the fusion confidence. The application can improve the stability and safety of unmanned aerial vehicle target tracking under visual occlusion, rotor noise and complex environment.

Description

Technical Field

[0001] This invention relates to the field of unmanned aerial vehicle (UAV) target tracking technology, and in particular to an audio-visual fusion UAV target tracking method and system based on rotor speed-driven harmonic suppression. Background Technology

[0002] Unmanned aerial vehicle (UAV) target tracking typically relies on visual detection and tracking technologies. A typical visual tracking scheme uses a deep learning-based target detector to detect image frames, calculates the target azimuth angle based on the detection box position, and inputs the detection results of consecutive frames into the tracking algorithm to maintain lock. However, in situations such as occlusion, drastic changes in lighting, cluttered backgrounds, or motion blur, the confidence and positioning accuracy of visual detection decrease significantly or even fail completely. Tracking systems that rely solely on a single visual modality are not robust enough in degraded scenarios.

[0003] Microphone array-based sound source localization technology can provide location information independent of illumination. Among them, the GCC-PHAT algorithm improves the sharpness of time delay estimation by using phase whitening of the cross power spectrum and is a widely used sound source localization method. However, the blade passing frequency and harmonics generated by the propeller rotation on the UAV platform can seriously contaminate the microphone signal, resulting in increased location estimation error. Existing solutions mostly use fixed-band notch filtering or adaptive noise cancellation to suppress the above self-noise, but fixed-band notch filtering cannot follow the rotation speed change, while adaptive noise cancellation has high computational complexity, both of which are not conducive to embedded real-time deployment.

[0004] Existing technologies using a single sensing mode struggle to provide stable and reliable orientation estimation independently in complex environments. Therefore, multimodal fusion has become a crucial technology for UAV target tracking. However, existing fusion schemes still have shortcomings in quantifying and utilizing the reliability of multimodal observations.

[0005] Firstly, some existing solutions set fixed fusion weights for acoustic and visual observations during the system initialization phase, which cannot dynamically reflect changes in the reliability of each mode according to different operating conditions. When the positioning error of the visual mode increases due to occlusion, backlighting, or motion blur, the erroneous observations still participate in the fusion with fixed weights and affect the overall output. Similarly, when the acoustic mode is affected by environmental noise or rotor harmonic interference, the system cannot automatically reduce its impact, resulting in a decrease in the stability of the fusion results.

[0006] Secondly, some existing solutions set an observation quality threshold, and when a certain mode falls below the threshold, they directly switch to another mode for target tracking. Although this method can maintain system operation when a single mode fails severely, it is prone to frequent switching when the observation quality fluctuates slightly near the threshold due to the hard switching mechanism. This causes jumps in target azimuth output, trajectory jitter, and system instability. At the same time, the hard switching method directly discards some effective information in the degraded mode, resulting in insufficient information utilization.

[0007] Third, some existing solutions use scalar indicators such as target detection confidence, signal-to-noise ratio, or energy value as fusion weights to linearly weight multimodal observations. Although this method can achieve a certain degree of dynamic fusion, the dynamic range of the weights is limited, and it is difficult to fully suppress abnormal observations in extreme degradation scenarios. At the same time, the classification confidence output by the detector usually reflects the probability of target category recognition, rather than the real uncertainty at the localization level. It is difficult to accurately describe the distribution of observation error, which can easily lead to deviations between the fusion results and the actual target location.

[0008] Furthermore, Chinese patent application CN115902786A discloses an acoustic tracking and countermeasure interception device, system, and method for intruding drones. It combines the YOLO-v5 algorithm with a sound source localization system to achieve synchronous tracking and countermeasure control. However, its focus is on the interception of intruding drones by fixed or ground-based countermeasure equipment. It does not establish a suppression mask for the dynamic harmonic self-noise caused by the rotor speed of the drone itself, nor does it convert acoustic and visual uncertainties into observation covariance scaling, gating, and flight control confidence. Chinese patent application CN116008913A discloses a drone detection and localization method based on a small microphone array. It uses the GCC-PHAT algorithm for time delay estimation and solves for sound source coordinates. However, it focuses on acoustic single-mode detection and localization, does not utilize real-time rotor speed sequences to suppress rotor harmonics, and does not form a closed-loop collaboration with visual cognitive uncertainty, interactive multi-model filters, and control state machines.

[0009] Although there have been preliminary explorations in audio-visual fusion for uncertainty perception in academic research, such as COLDFusion dynamically characterizing the information content of different modalities by learning the potential distribution of audio and visual modalities, this type of research is usually geared towards classification or regression tasks such as emotion recognition and speech recognition, and has not been specifically designed for rotor harmonic self-noise, azimuth state estimation, flight control safety degradation and multi-motion model interactive filtering of UAV platforms. Summary of the Invention

[0010] In view of the shortcomings of the prior art, the present invention provides an audio-visual fusion UAV target tracking method and system based on rotor speed-driven harmonic suppression, in order to solve the problems of insufficient tracking stability in complex degradation scenarios caused by the lack of rotor harmonic suppression mechanism that varies with rotational speed, the lack of probabilistic uncertainty modeling for the reliability of acoustic and visual observations, and the inability of the fusion strategy to continuously transmit uncertainty to state estimation and flight control.

[0011] To achieve the above objectives, the present invention adopts the following technical solution:

[0012] In a first aspect, the present invention provides a method for audio-visual fusion UAV target tracking based on rotor speed-driven harmonic suppression, comprising:

[0013] A. Acquire multimodal data;

[0014] The multimodal data includes multi-channel microphone data, image data of the target to be tracked, and rotor speed data of the UAV itself;

[0015] B. Calculate the blade passage frequency based on the rotor speed data of the UAV, and generate a harmonic suppression mask according to the blade passage frequency to suppress the harmonic noise of the UAV rotor.

[0016] C. Calculate the cross-power spectrum and signal-to-noise ratio adaptive weight of the multi-channel microphone data, and perform weighted processing on the cross-power spectrum according to the harmonic suppression mask and the signal-to-noise ratio adaptive weight to generate a weighted cross-correlation spectrum to estimate acoustic orientation and acoustic uncertainty.

[0017] D. Perform target detection on the image data of the target to be tracked to determine the visual orientation of the target, and generate visual uncertainty through a pre-trained evidence regression model;

[0018] E. Calculate the acoustic observation noise covariance and the visual observation noise covariance dynamically based on the acoustic uncertainty and the visual uncertainty, respectively;

[0019] F. Determine whether the acoustic uncertainty and / or the visual uncertainty exceed the corresponding preset gating threshold;

[0020] If the number of observations exceeds the limit, the corresponding acoustic and / or visual orientations will be marked as invalid observations and will not be included in subsequent measurement updates.

[0021] If the observation is not exceeded, the corresponding acoustic and / or visual orientation is marked as a valid observation, and step G is executed.

[0022] G. Based on the multi-model state estimation filter, sequential measurement updates are performed on the effective observations and their corresponding observation noise covariance in a preset order to output the fused orientation and fused covariance.

[0023] H. Calculate the fusion confidence score based on the fusion covariance, and adaptively scale the control gain according to the fusion orientation and the fusion confidence score to generate tracking control commands to control the UAV to track the target.

[0024] In an optional implementation, the function expression for the frequency of the blade passage is:

[0025]

[0026] In the formula, The frequency at which the blades pass through; This refers to the rotor speed data of the drone itself. This refers to the number of rotor blades;

[0027] The functional expression for the harmonic suppression mask is:

[0028]

[0029] In the formula, This is a harmonic suppression mask; H is the highest harmonic order; h is the harmonic order; The soft mask depth is the h-th harmonic. For frequency; denoted as the suppression bandwidth of the h-th harmonic.

[0030] In an optional implementation, step C specifically includes the following steps:

[0031] The multi-channel microphone data is sampled according to a time window and a fast Fourier transform is performed to obtain the frequency domain representation of each channel;

[0032] The multi-channel microphone signal is acquired by a multi-channel microphone array deployed on the drone, and the multi-channel microphone signal is a time-domain audio signal.

[0033] Based on the multi-channel microphone array, several microphone pairs are constructed, and the phase transformation whitening cross power spectrum of each microphone pair is calculated.

[0034] Based on the frequency domain representation of each channel, the signal power of each frequency point in the frequency domain representation of each channel is calculated, and the noise power of each frequency point is estimated according to the power spectrum low quantile filtering or minimum statistical method.

[0035] Based on the signal power and noise power of each frequency point, the signal-to-noise ratio (SNR) of each frequency point is calculated, and an adaptive SNR weight is generated according to the SNR.

[0036] The phase-transformed whitened cross-power spectrum, the harmonic suppression mask, and the signal-to-noise ratio adaptive weights are multiplied point-by-point in the frequency domain and then subjected to inverse Fourier transform to obtain the weighted cross-correlation spectrum.

[0037] The time delay search domain is determined based on the spacing and sound speed of the microphone pair. Within the time delay search domain, the peak position of the weighted cross-correlation spectrum is used as the estimated time delay, and the estimated time delay mapping is used as the acoustic azimuth angle of the microphone pair.

[0038] The acoustic azimuth angles of each microphone pair are circularly statistically aggregated, and the weighted sine and cosine sums are calculated to obtain the average synthesized vector length. The complement of the average synthesized vector length is used as the acoustic uncertainty.

[0039] In an optional implementation, the functional expression of the phase-transform whitening cross-power spectrum is:

[0040]

[0041] In the formula, Phase-transformed whitened cross-power spectrum; For the i-th microphone in the microphone pair; Let be the complex conjugate of the j-th microphone in the microphone pair;

[0042] The functional expression for the signal-to-noise ratio adaptive weight is:

[0043]

[0044] In the formula, For signal-to-noise ratio adaptive weights; It is an S-shaped function; The slope coefficient of the sigmoid function; The signal-to-noise ratio at frequency f is expressed in dB. For reference signal-to-noise ratio;

[0045] The functional expression for the weighted cross-correlation spectrum is:

[0046]

[0047] In the formula, For weighted cross-correlation spectrum; For time delay variables; For harmonic suppression mask; This is the inverse fast Fourier transform;

[0048] The functional expression for the acoustic azimuth angle is:

[0049]

[0050] In the formula, Acoustic azimuth; To estimate the time delay; Speed ​​of sound; The spacing between the microphone pairs; This is a truncation function.

[0051] In an optional implementation, step D specifically includes the following steps:

[0052] Perform target detection on the image data of the target to be tracked to obtain the coordinates of the detection box and the center point of the detection box;

[0053] The image data of the target to be tracked is collected by the camera mounted on the drone;

[0054] The visual azimuth angle is calculated based on the center point of the detection frame and the pre-calibrated camera intrinsic parameters;

[0055] The target image data is input into a pre-trained evidence regression model to obtain several normal-inverse gamma distribution parameters, and the visual uncertainty is calculated based on the several distribution parameters.

[0056] The functional expression for the visual azimuth angle is:

[0057]

[0058] In the formula, Visual azimuth angle; The horizontal pixel coordinates of the center point of the detection box; The x-coordinate of the principal point of the camera; The horizontal focal length of the camera;

[0059] The functional expression for the visual uncertainty is:

[0060]

[0061] In the formula, For visual uncertainty; , , All parameters are normal-inverse gamma distribution parameters.

[0062] In an optional implementation, step E specifically includes the following steps:

[0063] Using the acoustic uncertainty as an exponential scaling factor, the preset acoustic observation noise covariance base value is scaled first, and the first scaling result is truncated with upper and lower limits to obtain the acoustic observation noise covariance.

[0064] Using the visual uncertainty as an exponential scaling factor, the preset visual observation noise covariance base value is scaled a second time, and the upper and lower limits of the second scaling result are truncated to obtain the visual observation noise covariance.

[0065] The functional expression for the acoustic observation noise covariance is:

[0066]

[0067] In the formula, For acoustic observation noise covariance; The base value for the noise covariance of acoustic observations; is the scaling factor for acoustic uncertainty on the acoustic observation noise covariance; Acoustic uncertainty;

[0068] The functional expression for the visual observation noise covariance is:

[0069]

[0070] In the formula, For visual observation noise covariance; The basic value of visual observation noise covariance; This is the scaling factor for visual uncertainty on the visual observation noise covariance; This represents visual uncertainty.

[0071] In an optional implementation, step G specifically includes the following steps:

[0072] Based on the preset model transition probability matrix, interactive mixing is performed on each sub-model in the multi-model state estimation filter to obtain the mixed initial state and mixed initial covariance of each sub-model.

[0073] Starting from the initial mixed state and the initial mixed covariance of each sub-model, extended Kalman filter prediction is performed on each sub-model to obtain the prediction state and prediction covariance of each sub-model.

[0074] Based on the predicted state and predicted covariance of each sub-model, sequential measurement updates are performed on the effective observations to obtain the innovation and innovation covariance of each sub-model, and the updated state estimate and updated covariance of each sub-model are determined according to the innovation and innovation covariance.

[0075] Based on the innovation and innovation covariance of each sub-model, calculate the likelihood value of each sub-model and update the model probability of each sub-model;

[0076] The updated state estimate and the updated covariance are weighted and fused according to the model probability to obtain the fused orientation and the fused covariance.

[0077] When both the acoustic orientation and the visual orientation are marked as invalid observations, the sequential measurement update is not performed. Instead, the predicted state of each sub-model is used as the state estimate of each sub-model, and weighted fusion is performed.

[0078] In an optional implementation, step H specifically includes the following steps:

[0079] The fusion confidence is calculated based on the fusion covariance, and the azimuth deviation is calculated based on the fusion azimuth.

[0080] The azimuth deviation is the difference between the target's azimuth and the UAV's current orientation;

[0081] The PID control gain is adaptively scaled based on the fusion confidence level to obtain the scaled PID control gain.

[0082] The PID control gain includes proportional gain, integral gain, and derivative gain;

[0083] When the fusion confidence level is lower than the preset confidence threshold, or when both the acoustic orientation and the visual orientation are marked as invalid observations, the control gain is attenuated or the integral term in the control gain is frozen, and the UAV is controlled to switch to a degraded state or a return-to-home state.

[0084] When the fusion confidence is not lower than the preset confidence threshold, and the acoustic orientation and the visual orientation are not all marked as invalid observations, the tracking control command is generated based on the orientation deviation and the scaled control gain.

[0085] Secondly, the present invention provides an audio-visual fusion UAV target tracking system based on rotor speed-driven harmonic suppression, comprising:

[0086] The data acquisition module is used to acquire multimodal data;

[0087] The multimodal data includes multi-channel microphone data, image data of the target to be tracked, and rotor speed data of the UAV itself;

[0088] The mask generation module is used to calculate the blade passing frequency based on the rotor speed data of the UAV and generate a harmonic suppression mask according to the blade passing frequency to suppress the harmonic noise of the UAV rotor.

[0089] The acoustic sensing module is used to calculate the cross power spectrum and signal-to-noise ratio adaptive weight of the multi-channel microphone data, and to perform weighted processing on the cross power spectrum according to the harmonic suppression mask and the signal-to-noise ratio adaptive weight to generate a weighted cross-correlation spectrum to estimate the acoustic orientation and acoustic uncertainty.

[0090] The visual perception module is used to perform target detection on the image data of the target to be tracked, so as to determine the visual orientation of the target to be tracked, and to generate visual uncertainty through a pre-trained evidence regression model.

[0091] The covariance determination module is used to dynamically calculate the acoustic observation noise covariance and the visual observation noise covariance based on the acoustic uncertainty and the visual uncertainty, respectively.

[0092] The gating judgment module is used to determine whether the acoustic uncertainty and / or the visual uncertainty exceed the corresponding preset gating threshold.

[0093] If the number of observations exceeds the limit, the corresponding acoustic and / or visual orientations will be marked as invalid observations and will not be included in subsequent measurement updates.

[0094] If the observation is not exceeded, the corresponding acoustic and / or visual orientation is marked as a valid observation, and the data fusion module is executed.

[0095] The data fusion module is used to perform sequential measurement updates on the effective observations and their corresponding observation noise covariance according to a preset order based on a multi-model state estimation filter, so as to output the fused orientation and fused covariance.

[0096] The UAV control module is used to calculate the fusion confidence based on the fusion covariance, and adaptively scale the control gain according to the fusion orientation and the fusion confidence to generate tracking control commands to control the UAV to track the target.

[0097] Thirdly, the present invention provides an electronic device, characterized in that it comprises:

[0098] Sensor components are used to acquire multimodal data;

[0099] The sensor assembly includes a multi-channel microphone array, a camera, and a speed sensor;

[0100] The processor is connected to the sensor assembly;

[0101] A memory, connected to the processor, for storing a computer program, wherein the processor executes the computer program to implement the method as described in the first aspect;

[0102] A communication interface is provided for connection to the processor.

[0103] The beneficial effects of the embodiments provided by the present invention include:

[0104] This invention modifies the acoustic positioning process by using rotor speed-driven harmonic suppression and adaptive signal-to-noise ratio (SNR) weighting. It accurately suppresses rotor harmonic noise by using a frequency domain mask that varies with the rotational speed, and enhances the contribution of high SNR frequency bands by using adaptive SNR weighting. This solves the problems of fixed-band notch filtering not following rotational speed changes and the computational complexity of adaptive noise cancellation. It can still obtain stable acoustic orientation estimation in scenarios with changing flight conditions and increased environmental noise.

[0105] This invention uses probabilistic modeling of acoustic and visual uncertainties to quantify the reliability of multimodal observations into a continuous uncertainty measure. Based on this, the observation noise covariance of each modality is dynamically determined and gating operations are performed. The impact of unreliable observations on the fusion state is continuously attenuated until it is completely blocked. This avoids the insufficient environmental adaptability of fixed-weight fusion, the frequent jumps of hard-switching fusion, and the insufficient expression of uncertainty by linear weighted scalar confidence.

[0106] This invention continuously transmits uncertainty information from the perception layer to the control layer through the fusion layer. The fusion confidence is calculated by the fusion covariance and the control gain is adaptively scaled. When the fusion result is uncertain, the control strength is automatically reduced or the state degradation is triggered. This realizes a closed loop of uncertainty in the entire link from perception, fusion to flight control, which significantly improves the stability and safety of UAV target tracking in visual occlusion, rotor noise and complex environments. Attached Figure Description

[0107] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0108] Figure 1 A flowchart illustrating the audio-visual fusion UAV target tracking method based on rotor speed-driven harmonic suppression in an embodiment of this specification is shown.

[0109] Figure 2 A schematic diagram of the signal processing flow of the acoustic modes in the embodiments of this specification is shown;

[0110] Figure 3 This document illustrates a schematic diagram of the multimodal interaction, multi-model fusion, and adaptive control process in an embodiment of this specification.

[0111] Figure 4 This specification illustrates a schematic diagram of the finite state transition process for UAV tracking in an embodiment of the specification.

[0112] Figure 5 A block diagram of an audio-visual fusion UAV target tracking system based on rotor speed-driven harmonic suppression, as shown in the embodiments of this specification, is illustrated. Detailed Implementation

[0113] The features and exemplary embodiments of various aspects of the present invention will now be described in detail. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that the invention may be practiced without requiring some of these specific details. The following description of embodiments is merely intended to provide a better understanding of the invention by illustrating examples of the invention.

[0114] In some of the processes described in the specification, claims, and accompanying drawings of this invention, multiple operations appear in a specific order. However, it should be clearly understood that these operations may not be performed in the order they appear herein, or they may be performed in parallel. The operation numbers, such as S1, S2, etc., are merely used to distinguish different operations and do not themselves represent any execution order. Furthermore, these processes may include more or fewer operations, and these operations may be performed sequentially or in parallel.

[0115] Example 1

[0116] like Figures 1-3 As shown, this embodiment provides a sound-visual fusion UAV target tracking method based on rotor speed-driven harmonic suppression, including:

[0117] S1. Acquire multimodal data;

[0118] In some embodiments, the multimodal data is acquired by sensor components mounted on the drone; the sensor components include a multichannel microphone array, a camera, and a rotation sensor;

[0119] The multimodal data includes multi-channel microphone data, image data of the target to be tracked, and rotor speed data of the UAV itself.

[0120] Specifically, a multi-channel microphone array can be installed at the end of the UAV's arm to maximize the array baseline and move away from the propeller noise source for collecting multi-channel microphone data; a camera can be installed on the UAV's nose or gimbal for collecting image data of the target to be tracked; the UAV's rotor speed data can be collected by a speed sensor or obtained through the UAV's flight controller interface or electronic speed controller interface.

[0121] In this embodiment, the multi-channel microphone data is the time-domain acoustic signal synchronously acquired by each microphone channel; the target image data to be tracked is a series of image frames containing the target to be tracked; and the rotor speed data of the UAV is the real-time speed sequence of each motor.

[0122] S2. Calculate the blade passage frequency based on the rotor speed data of the UAV, and generate a harmonic suppression mask according to the blade passage frequency to suppress the harmonic noise of the UAV rotor.

[0123] Specifically, the function expression for the frequency of the blade passage is:

[0124]

[0125] In the formula, The frequency at which the blades pass through; This refers to the rotor speed data of the drone itself. This refers to the number of rotor blades;

[0126] In this embodiment, the harmonic suppression mask adopts a soft suppression form, which applies suppression that attenuates as the frequency offset increases to the frequency band near each order rotor harmonic, and the calculated harmonic suppression mask is truncated to [0,1] as the final harmonic suppression mask;

[0127] The functional expression for the harmonic suppression mask is:

[0128]

[0129] In the formula, This is a harmonic suppression mask; H is the highest harmonic order; h is the harmonic order; The soft mask depth is the h-th harmonic. For frequency; The suppression bandwidth of the h-th harmonic;

[0130] In some embodiments, the harmonic suppression mask can also be in binary form, and its functional expression is as follows:

[0131]

[0132] In the formula, 0 indicates that the signal components within the frequency band are suppressed; 1 indicates that the signal components outside the frequency band are preserved.

[0133] In a preferred embodiment, when rotor speed data is missing, abnormally jumps, or exceeds the time limit, the harmonic suppression mask is stopped, and the acoustic uncertainty is set to a preset high value to trigger subsequent gating. Specifically, when rotor speed data is missing, abnormally jumps, or exceeds the time threshold, the speed data is no longer used to generate the harmonic suppression mask, and the acoustic uncertainty is set to a preset high value higher than the acoustic gating threshold. This causes the acoustic observation to be marked as an invalid observation in subsequent gating judgments and not to participate in subsequent measurement updates, thereby preventing abnormal speed data from causing erroneous acoustic observations to enter the fusion filtering and control closed loop.

[0134] S3. Calculate the cross-power spectrum and signal-to-noise ratio adaptive weight of the multi-channel microphone data, and perform weighted processing on the cross-power spectrum according to the harmonic suppression mask and the signal-to-noise ratio adaptive weight to generate a weighted cross-correlation spectrum to estimate acoustic orientation and acoustic uncertainty.

[0135] For example, step S3 specifically includes the following steps:

[0136] S301. Sample the multi-channel microphone data according to a time window and perform a fast Fourier transform to obtain the frequency domain representation of each channel;

[0137] S302. Based on the microphones in the multi-channel microphone array, construct several microphone pairs and calculate the phase transformation whitening cross power spectrum of each microphone pair.

[0138] The functional expression for the phase-transformed whitening cross-power spectrum is:

[0139]

[0140] In the formula, Phase-transformed whitened cross-power spectrum; For the i-th microphone in the microphone pair; Let be the complex conjugate of the j-th microphone in the microphone pair;

[0141] S303. Based on the frequency domain representation of each channel, calculate the signal power of each frequency point in the frequency domain representation of each channel, and estimate the noise power of each frequency point according to the power spectrum low quantile filtering or minimum statistical method.

[0142] S304. Based on the signal power and noise power of each frequency point, calculate the signal-to-noise ratio of each frequency point, and generate an adaptive signal-to-noise ratio weight according to the signal-to-noise ratio.

[0143] Specifically, the functional expression for the signal-to-noise ratio is:

[0144]

[0145] In the formula, Signal-to-noise ratio; The signal power at each frequency point; The noise power at each frequency point;

[0146] Specifically, the functional expression for the signal-to-noise ratio adaptive weights is:

[0147]

[0148] In the formula, For signal-to-noise ratio adaptive weights; It is an S-shaped function; The slope coefficient of the sigmoid function is used to control the sensitivity of the signal-to-noise ratio adaptive weight to changes in the signal-to-noise ratio. The signal-to-noise ratio at frequency f is expressed in dB. For reference signal-to-noise ratio;

[0149] Among them, the signal-to-noise ratio adaptive weight approaches one at frequencies with high signal-to-noise ratio and approaches zero at frequencies with low signal-to-noise ratio;

[0150] Preferably, this embodiment satisfies ,when When the signal-to-noise ratio (SNR) is higher than the reference SNR threshold, the adaptive SNR weight approaches 1, increasing the contribution of that frequency point to the cross-correlation spectrum; when... When the signal-to-noise ratio is below the reference signal-to-noise ratio threshold, the adaptive signal-to-noise ratio weight approaches 0, thus suppressing low signal-to-noise ratio frequencies.

[0151] In some embodiments, the functional expression for the signal-to-noise ratio adaptive weights is:

[0152]

[0153] S305. Perform an inverse Fourier transform after multiplying the phase transform whitened cross power spectrum, the harmonic suppression mask and the signal-to-noise ratio adaptive weights point by point in the frequency domain to obtain the weighted cross-correlation spectrum.

[0154] Among them, the weighted cross-correlation spectrum is used to reflect the degree of correlation between the two signals under different time delays;

[0155] The functional expression for the weighted cross-correlation spectrum is:

[0156]

[0157] In the formula, For weighted cross-correlation spectrum; For time delay variables; For harmonic suppression mask; This is the inverse fast Fourier transform;

[0158] S306. Determine the time delay search domain based on the spacing and sound speed of the microphone pair. Within the time delay search domain, use the peak position of the weighted cross-correlation spectrum as the estimated time delay, and use the estimated time delay mapping as the acoustic azimuth angle of the microphone pair.

[0159] Specifically, in restricting the search domain Within the range, the peak position of the weighted cross-correlation spectrum is taken, and parabolic interpolation is used between the peak and adjacent sample points to obtain the sub-sampling level estimated time delay;

[0160] The function expression for estimating the time delay is:

[0161]

[0162] In the formula, To estimate the time delay;

[0163] The functional expression for the acoustic azimuth angle is:

[0164]

[0165] In the formula, Acoustic azimuth; Speed ​​of sound; The spacing between the microphone pairs; This is a truncation function used to truncate the parameter to the interval [–1, 1].

[0166] S307. Perform circular statistical aggregation on the acoustic azimuth angles of each microphone pair, and calculate the weighted sine and weighted cosine sums to obtain the average synthesized vector length. Use the complement of the average synthesized vector length as the acoustic uncertainty.

[0167] Specifically, the functional expressions for the weighted sine sum and the weighted cosine sum are as follows:

[0168]

[0169]

[0170] In the formula, For weighted sine sum; For weighted cosine sum; The weights of the microphone pair including the i-th and j-th microphones;

[0171] Specifically, the functional expression for the average composite vector length is:

[0172]

[0173] In the formula, This represents the average length of the synthesized vector.

[0174] Specifically, the functional expression for acoustic uncertainty is:

[0175]

[0176] In the formula, Acoustic uncertainty;

[0177] It should be noted that when the acoustic azimuth angles of each microphone pair are consistent, the average synthesized vector length approaches one, and the acoustic uncertainty approaches zero, indicating that the acoustic positioning is reliable; when the acoustic azimuth angles are discrete, the average synthesized vector length approaches zero, and the acoustic uncertainty approaches one, indicating that the acoustic positioning is unreliable.

[0178] In this embodiment, the target acoustic azimuth is extracted from multi-channel microphone data contaminated by rotor harmonic noise and environmental noise, and the acoustic uncertainty characterizing the reliability of the azimuth estimation is output simultaneously. This embodiment eliminates the influence of signal amplitude through phase transformation whitening, while retaining the phase information crucial for time delay estimation. Adaptive weights are generated based on signal-to-noise ratio (SNR) estimation, enabling higher contribution from high SNR frequency bands. Harmonic suppression masks are combined with SNR adaptive weights for cross-power spectrum to achieve dual suppression of rotor harmonic noise and environmental noise, resulting in an enhanced weighted cross-correlation spectrum. Azimuth angle estimates from multiple microphone pairs are aggregated through circular statistical methods, and the consistency of each azimuth angle is quantified by the complement of the average synthesized vector length, yielding the acoustic uncertainty, which can quantitatively assess the reliability of its own acoustic positioning.

[0179] S4. Perform target detection on the image data of the target to be tracked to determine the visual orientation of the target, and generate visual uncertainty through a pre-trained evidence regression model.

[0180] For example, step S4 specifically includes the following steps:

[0181] S401. Perform target detection on the image data of the target to be tracked to obtain the coordinates of the detection box and the center point of the detection box;

[0182] S402. Calculate the visual azimuth angle based on the center point of the detection frame and the pre-calibrated camera intrinsic parameters;

[0183] The functional expression for the visual azimuth angle is:

[0184]

[0185] In the formula, Visual azimuth angle; The horizontal pixel coordinates of the center point of the detection box; The x-coordinate of the principal point of the camera; The horizontal focal length of the camera;

[0186] S403. Input the target image data into a pre-trained evidence regression model to obtain several normal-inverse gamma distribution parameters, and calculate the visual uncertainty based on the several distribution parameters;

[0187] Specifically, the functional expression for visual uncertainty is:

[0188]

[0189] In the formula, For visual uncertainty; , , All are parameters of a normal-inverse gamma distribution, where, The confidence strength parameter for predicting the mean; The shape parameter of the inverse gamma distribution; The scaling parameter for the inverse gamma distribution;

[0190] In some embodiments, the evidence regression model consists of a target detection backbone feature network and an evidence regression head attached thereto, which outputs both orientation estimation and cognitive uncertainty in a single forward propagation without the need for multiple Monte Carlo samplings.

[0191] The evidence regression head outputs four parameters of a normal-inverse gamma distribution. , , , ,in As an azimuth point estimate This reflects the model's confidence in the predicted mean. , Let be the shape and scale parameters of the inverse gamma distribution, and , 1, ;middle

[0192] In this embodiment, when the target image data belongs to the training sufficiently covered area, Larger Larger The visual uncertainty tends to be smaller when the target image data belongs to an out-of-distribution or sparse sample region. Smaller The larger the value, the greater the visual uncertainty tends to be;

[0193] In an optional embodiment, the target detection backbone feature network can be the backbone network of detectors such as YOLO series or SSD, used to extract depth visual features from the input target image data; the evidence regression head is connected to the backbone network and used to output parameters of a normal-inverse gamma distribution based on the depth visual features.

[0194] Specifically, the probabilistic modeling method of the evidence regression model is as follows:

[0195] Assume the predicted azimuth angle y follows a normal distribution. For the unknown mean and variance Apply a normal-inverse gamma prior distribution, i.e.:

[0196]

[0197] In the formula, The mean and variance Normal-inverse gamma joint prior probability density; For variance Mean under given conditions The conditional normal distribution; For variance The inverse gamma distribution; This refers to the predicted mean of the azimuth angle or location parameters.

[0198] Among them, the pre-training of the evidence regression model adopts the method of joint optimization of the negative log marginal likelihood loss function and the regularization loss function based on the normal-inverse gamma prior, so that the model outputs lower uncertainty in the training coverage area and higher uncertainty in the off-distribution or sparse sample area.

[0199] The negative log-marginal likelihood loss is derived from the Student-t marginal likelihood and is equivalent to the negative log-marginal likelihood of the predicted value y when the prior is a normal-inverse gamma distribution and the likelihood is a normal distribution. Its functional expression is:

[0200]

[0201] In the formula, The negative logarithmic marginal likelihood loss; For the gamma function in parameters The function value at that location;

[0202] Among them, the gamma function is a generalization of the factorial function in the real number field, used to calculate the normalization term of the Student-t marginal distribution in the negative logarithmic marginal likelihood loss.

[0203] Regularization loss is used to penalize models that still output excessive evidence when predictions significantly deviate from the labels. Its functional expression is:

[0204]

[0205] In the formula, This is the regularization loss; For the true azimuth angle label Compared with the predicted mean The absolute error between them; As the evidence level, when the prediction error is large but the evidence level is still high, the regularization loss is increased to impose a penalty;

[0206] The total loss function is:

[0207]

[0208] In the formula, This is the total loss function; The preset regularization coefficient is used to balance the weights between the negative logarithmic marginal likelihood loss and the regularization loss.

[0209] By using the aforementioned loss function to perform gradient backpropagation and iterative optimization of the model parameters, the model can achieve higher outputs in regions where the training is sufficiently covered. and Smaller This results in less visual uncertainty; and lower output in out-of-distribution or sparse sample regions. higher This results in less visual uncertainty; thus, the evidence regression model has the ability to perceive the cognitive uncertainty of its own predictive reliability.

[0210] In an optional embodiment, step S4 further includes an exception handling mechanism:

[0211] When the detection box is not detected, or the detection box is not filtered by the region of interest, the visual orientation is marked as invalid, and the visual uncertainty is set to a value above the preset gating threshold to trigger subsequent gating skip.

[0212] The region of interest filtering includes at least one of the following:

[0213] The area ratio of the detection box is outside the preset area ratio range; the aspect ratio of the detection box is outside the preset aspect ratio range; the distance between the detection box and the image edge is less than the preset edge margin.

[0214] When multiple detection boxes exist in a frame of target image data, a unique detection box is selected from the multiple detection boxes based on at least one of the following criteria: detection confidence, detection box area, angular distance from the predicted azimuth angle at the previous moment, or similarity of appearance features, for the purpose of calculating the visual orientation.

[0215] In this embodiment, all visual azimuth-related errors are mapped to ( ) using an azimuth normalization function. [Interval, to avoid numerical jumps near ±π.]

[0216] S5. Based on the acoustic uncertainty and the visual uncertainty, dynamically calculate the acoustic observation noise covariance and the visual observation noise covariance respectively;

[0217] In some embodiments, the functional expression for the acoustic observation noise covariance is:

[0218]

[0219] In the formula, For acoustic observation noise covariance; The preset acoustic observation noise covariance baseline value; Preset acoustic scaling factor; Acoustic uncertainty;

[0220] In some embodiments, the functional expression for the visual observation noise covariance is:

[0221]

[0222] In the formula, For visual observation noise covariance; The preset visual observation noise covariance baseline value; Preset visual scaling factor; For visual uncertainty;

[0223] S6. Determine whether the acoustic uncertainty and / or the visual uncertainty exceed the corresponding preset gating threshold;

[0224] If the number of observations exceeds the limit, the corresponding acoustic and / or visual orientations will be marked as invalid observations and will not be included in subsequent measurement updates.

[0225] If the observation is not exceeded, the corresponding acoustic and / or visual orientation is marked as a valid observation, and step S7 is executed.

[0226] S7. Based on the multi-model state estimation filter, perform sequential measurement updates on the effective observations and their corresponding observation noise covariance in a preset order to output the fused orientation and fused covariance.

[0227] For example, step S7 specifically includes the following steps:

[0228] S701. Based on the preset model transition probability matrix, perform interactive mixing on each sub-model in the multi-model state estimation filter to obtain the mixed initial state and mixed initial covariance of each sub-model.

[0229] The model transition probability matrix can be preset according to the relative motion pattern between the UAV and the tracked target. The main diagonal elements of the model transition probability matrix represent the probability that the current motion pattern of each sub-model remains unchanged, and the off-diagonal elements of the model transition probability matrix represent the switching probability between different motion patterns.

[0230] In some embodiments, the multi-model state estimation filter is an interactive multi-model extended Kalman filter, and the sub-models in the filter include a uniform velocity model, a uniform acceleration model, and a coordinated turning model.

[0231] Specifically, the state vectors of each sub-model are as follows:

[0232]

[0233]

[0234] In the formula, This is the state vector of the constant velocity model; This represents the state vector of the constant acceleration model. To coordinate the state vectors of the turning model; The azimuth angle of the target relative to the drone; This represents the rate of change of azimuth angle; It is the azimuth angular acceleration; To coordinate the turning rate in the turning model;

[0235] Specifically, the state transition matrices for each sub-model are as follows:

[0236]

[0237]

[0238] In the formula, This is the state transition matrix of the constant-rate model; This is the state transition matrix of the constant acceleration model; The sampling time interval between two adjacent filtering moments;

[0239] Among them, the state transition matrix of the coordinated turning model is a nonlinear transition function with the turning rate as a parameter, and the extended Kalman filter linearizes it using its Jacobian matrix;

[0240] Specifically, the observation matrices for each sub-model are as follows:

[0241]

[0242]

[0243]

[0244] In the formula, This is the observation matrix for the constant velocity model; This represents the observation matrix of the constant acceleration model; To reconcile the observation matrix of the turning model;

[0245] In a preferred embodiment, the process noise covariance matrix of each sub-model is used to describe the impact of unmodeled disturbances or random accelerations on the uncertainty of state prediction during the target motion process.

[0246] The process noise covariance matrices of the constant acceleration motion model and the coordinated turning model can be obtained by using the same discretization method as the corresponding motion models. The process noise covariance matrix of the constant velocity motion model is obtained by discretizing the continuous white noise acceleration model, and its functional expression is as follows:

[0247]

[0248] In the formula, The process noise covariance matrix for the constant velocity motion model; For process noise intensity coefficients; matrix elements , and The time terms are obtained from the discretization of the continuous white noise acceleration model, respectively.

[0249] In some embodiments, interactive blending is performed according to the following function expression:

[0250]

[0251]

[0252]

[0253] In the formula, Given that the current model is j, the previous model was i. This represents the probability of transitioning from the i-th model to the j-th model in the model transition probability matrix; Let be the model probability of the i-th sub-model at the previous time step; Let be the normalization factor for the j-th sub-model; Let j be the initial state of the mixture for the j-th sub-model; This is the state estimate of the i-th sub-model at the previous time step; Let J be the initial covariance of the mixture for the j-th sub-model; The covariance of the state estimate for the i-th sub-model at the previous time step; This is the state estimate for the i-th sub-model;

[0254] S702. Starting from the initial mixed state and the initial mixed covariance of each sub-model, perform extended Kalman filter prediction on each sub-model to obtain the prediction state and prediction covariance of each sub-model.

[0255] Specifically, the extended Kalman filter prediction is performed according to the following function expression:

[0256]

[0257]

[0258] In the formula, The current state of the j-th sub-model is predicted from the information of the previous time step. Let $\mathbf{j}$ be the prediction covariance of the j-th sub-model, predicted using information from the previous time step. Let J be the state transition matrix of the j-th sub-model; Let J be the process noise covariance matrix of the j-th sub-model;

[0259] S703. Based on the predicted state and predicted covariance of each sub-model, perform sequential measurement updates on the effective observations to obtain the innovation and innovation covariance of each sub-model, and determine the updated state estimate and updated covariance of each sub-model according to the innovation and innovation covariance.

[0260] In some embodiments, sequential measurement updates are performed in the order of acoustic followed by visual, including two sub-steps: acoustic update and visual update.

[0261] Among them, acoustic update refers to performing the first extended Kalman filter measurement update on each sub-model based on the predicted state and prediction covariance of each sub-model, using the acoustic orientation marked as valid observation and its corresponding acoustic observation noise covariance, to obtain the intermediate state estimate and intermediate covariance after acoustic update, as well as acoustic innovation and acoustic innovation covariance.

[0262] Visual update refers to using the intermediate state estimates and intermediate covariance of each sub-model after acoustic update as a basis, and using the visual orientation marked as a valid observation and its corresponding visual observation noise covariance to perform a second extended Kalman filter measurement update on each sub-model to obtain the final state estimate and final covariance after visual update, as well as visual innovation and visual innovation covariance.

[0263] In this embodiment, if the acoustic orientation is marked as an invalid observation, the acoustic update is skipped, and the predicted state and prediction covariance of each sub-model are used as the intermediate state estimate and the intermediate covariance, directly proceeding to the visual update; if the visual orientation is marked as an invalid observation, the visual update is skipped, and the intermediate state estimate and the intermediate covariance are used as the final state estimate and the final covariance; if both the acoustic orientation and the visual orientation are marked as invalid observations, the sequential measurement update is skipped, and the predicted state and prediction covariance of each sub-model are used as the final state estimate and the final covariance.

[0264] Specifically, the acoustic update is performed according to the following function expression:

[0265]

[0266]

[0267]

[0268]

[0269] In the formula, The information covariance corresponding to acoustic measurements; This is the observation matrix of the current sub-model; For acoustic observation noise covariance; Small positive numbers used to improve numerical stability; It is the identity matrix; For acoustic measurements, the corresponding Kalman gain is used. It is the inverse matrix of the acoustic information covariance matrix; For target azimuth angle observation output by the acoustic branch; and These are the intermediate state estimates and intermediate covariance of the j-th sub-model after acoustic measurement updates; Angle normalization function;

[0270] Specifically, the acoustic update is performed according to the following function expression:

[0271]

[0272]

[0273]

[0274]

[0275] In the formula, The information covariance corresponding to visual measurements; For visual observation noise covariance; The Kalman gain corresponding to visual measurements; It is the inverse matrix of the visual information covariance matrix; The target azimuth angle observation output by the vision branch; and These are the state estimates and covariance of the j-th sub-model updated by acoustic and visual order measurements at the current time, respectively.

[0276] S704. Based on the innovation and innovation covariance of each sub-model, calculate the likelihood value of each sub-model and update the model probability of each sub-model.

[0277] Here, the model probability is the posterior probability of the true motion state of the tracked target under the effective observations at the current moment for each sub-model; at the same time, the model probability is also the weight of each sub-model participating in the final fusion; the higher the model probability, the better the sub-model matches the current effective observations, and the higher the proportion of its updated state estimate in the final fusion result.

[0278] Specifically, the likelihood values ​​of each sub-model are calculated using the Gaussian probability density function:

[0279]

[0280] In the formula, Let be the observation likelihood value of the j-th sub-model, used to measure the degree of matching between the prediction results of the sub-model and the current valid observations; This represents the information corresponding to the j-th sub-model, i.e., the difference between the actual observation and the predicted observation; Let J be the innovation covariance corresponding to the j-th sub-model; With zero mean and covariance The value of the Gaussian probability density function representing the new information;

[0281] Specifically, the model probabilities of each sub-model are updated according to the following function expression:

[0282]

[0283] In the formula, Let be the updated model probability of the j-th sub-model at the current time; l is the sub-model number in the normalized summation; This is the summation of the likelihood and normalization factor products over all sub-models;

[0284] When acoustic and visual observations are independent within the same time step, Acoustic likelihood Visual similarity The product of is obtained, that is When a certain mode is gated, the likelihood term corresponding to that mode is set to 1, so as to avoid invalid observations from having an undue impact on the model probability update.

[0285] S705. The updated state estimate and the updated covariance are weighted and fused according to the model probability to obtain the fused orientation and the fused covariance.

[0286] Specifically, the fusion state and the fusion covariance are calculated using the following functional expression:

[0287]

[0288]

[0289] In the formula, Estimate the fusion state at the current moment. The fusion covariance at the current moment; The outer product of the deviations between the state estimates of the j-th sub-model and the fused state estimates is used to reflect the degree of dispersion between the estimation results of different sub-models.

[0290] When both the acoustic orientation and the visual orientation are marked as invalid observations, the sequential measurement update is not performed, and the predicted state of each sub-model is used as the state estimate of each sub-model for weighted fusion.

[0291] S8. Calculate the fusion confidence based on the fusion covariance, and adaptively scale the control gain according to the fusion orientation and the fusion confidence to generate a tracking control command to control the UAV to track the target.

[0292] For example, S8 specifically includes the following steps:

[0293] S801. Calculate the fusion confidence based on the fusion covariance, and calculate the azimuth deviation based on the fusion azimuth.

[0294] Among them, the azimuth deviation is the difference between the target's azimuth and the UAV's current orientation;

[0295] Specifically, the fusion confidence score is calculated from the trace of the fusion covariance, and its functional expression is as follows:

[0296]

[0297] In the formula, To integrate confidence levels; These are preset coefficients; The trace of the matrix; To integrate covariance;

[0298] S802. Adaptively scale the PID control gain according to the fusion confidence level to obtain the scaled PID control gain.

[0299] The PID control gain includes proportional gain, integral gain, and derivative gain;

[0300] S803. When the fusion confidence level is lower than the preset confidence threshold, or when both the acoustic orientation and the visual orientation are marked as invalid observations, the control gain is attenuated or the integral term in the control gain is frozen, and the UAV is controlled to switch to a degraded state or a return-to-home state.

[0301] Specifically, the scaled control gains are as follows:

[0302]

[0303]

[0304]

[0305] In the formula, , , These are the scaled proportional gain, integral gain, and derivative gain, respectively. , , These are proportional gain, integral gain, and derivative gain, respectively.

[0306] S804. When the fusion confidence is not lower than the preset confidence threshold, and the acoustic orientation and the visual orientation are not all marked as invalid observations, the tracking control command is generated based on the orientation deviation and the scaled control gain.

[0307] Specifically, the tracking control command is the yaw rate command, and its function expression is:

[0308]

[0309]

[0310] In the formula, This is the yaw rate command; This is the orientation deviation; To integrate orientation; This indicates the drone's current orientation.

[0311] In a preferred embodiment, the method further includes an instruction gating step:

[0312] Before the tracking control command is sent to the flight controller, the command gating module verifies the tracking control command based on the preset maximum command rate, command timeliness, and link status.

[0313] If the transmission rate of the tracking control command exceeds the maximum command rate, or if the difference between the generation time of the tracking control command and the current time exceeds the command validity period, then the command is discarded.

[0314] If a link anomaly is detected, all commands should be prohibited from being issued to prevent loss of control.

[0315] In a preferred embodiment, the method further includes a sensor health monitoring step:

[0316] The generation time of the most recent valid observation for each mode is recorded. If the interval between the generation time of a valid observation for a certain mode and the current time exceeds the preset timeout threshold corresponding to that mode, then the mode is marked as unhealthy.

[0317] When all modes are marked as unhealthy, the finite state machine is triggered to switch to a degraded state or a return-to-home state.

[0318] When at least one mode recovers a valid observation within the timeout threshold, the finite state machine is triggered to perform automatic recovery.

[0319] like Figure 4 As shown, in some embodiments, the flight state of the drone tracking the target is managed by a finite state machine;

[0320] The finite state machine includes at least the search state, scan state, lock state, track state, lost state, degraded state, and return state.

[0321] Specifically, the meaning of each state and the switching conditions between states of the finite state machine are as follows:

[0322] In the search state, the UAV has not yet acquired the target's effective location information. When the first effective observation is detected, the finite state machine switches from the search state to the scanning state.

[0323] In the scanning state, the UAV has acquired preliminary orientation information but it is not yet stable. When there is a valid observation of any mode within a preset number of consecutive frames, the finite state machine switches from the scanning state to the locking state.

[0324] In the locked state, the UAV has acquired relatively stable orientation information but still needs further confirmation. When the visual modality is confirmed to be valid within a consecutive preset number of frames and the fusion uncertainty is lower than the preset locking threshold, the finite state machine switches from the locked state to the tracking state.

[0325] During the tracking state, the UAV stably tracks the target, and the tracking control command is generated and issued normally.

[0326] When the fusion confidence level is lower than a preset confidence threshold, the finite state machine switches from the tracking state to the degraded state; when all modalities are gated skipped for a preset number of frames, the finite state machine switches from the tracking state to the lost state.

[0327] In the lost state, if at least one mode of valid observation is recovered within a preset time, the finite state machine switches from the lost state to the search state; if it is not recovered after the preset time, the finite state machine switches from the lost state to the return state.

[0328] In the degraded state, the control gain is attenuated or the integral term is frozen. If the fusion confidence recovers to above the preset confidence threshold, the finite state machine switches from the degraded state to the tracking state. If the fusion confidence remains below the preset confidence threshold for a preset time, the finite state machine switches from the degraded state to the return-to-home state.

[0329] In the return-to-home state, the drone stops tracking the target and performs a return flight.

[0330] Example 2

[0331] This embodiment provides a sound-visual fusion UAV target tracking method based on rotor speed-driven harmonic suppression. The difference between this method and Embodiment 1 is that a thermal imaging perception step is added to supplement additional orientation observation information in visual degradation scenarios.

[0332] For example, the audio-visual fusion drone target tracking method also includes:

[0333] Acquire thermal imaging data;

[0334] The thermal imaging data is collected by a thermal imaging sensor mounted on the drone.

[0335] The thermal imaging data is normalized, and the normalized thermal imaging data is binarized using a preset luminance quantile as a threshold to obtain a binary hotspot mask.

[0336] Perform connected component analysis on the binary hotspot mask, select the connected region with the largest area as the target hotspot region, and discard connected regions with an area smaller than a preset area threshold.

[0337] The center horizontal pixel coordinates of the target hotspot area are mapped to the thermal imaging azimuth angle through the intrinsic parameters of the thermal imaging sensor.

[0338] Based on the target hotspot area, obtain the uncertainty of thermal imaging;

[0339] The thermal imaging uncertainty can be obtained based on at least one of the following methods: the ratio of the connected domain area of ​​the target hot spot region to the total bright area area, temperature contrast, or brightness variance.

[0340] In one optional implementation, the functional expression for the thermal imaging uncertainty is:

[0341] In the formula, Due to uncertainties in thermal imaging; The area of ​​the target hotspot region; The area of ​​all bright areas in the binary hotspot mask;

[0342] Based on the thermal imaging data, a thermal imaging orientation and thermal imaging uncertainty are generated, and the thermal imaging orientation and thermal imaging uncertainty are used as a third mode observation.

[0343] Based on the aforementioned thermal imaging uncertainty, the thermal imaging observation noise covariance is determined by exponential scaling.

[0344] In some embodiments, the functional expression for the thermal imaging observation noise covariance is:

[0345]

[0346] In the formula, For thermal imaging observation noise covariance; This is the preset baseline value for thermal imaging observation noise covariance; Preset thermal imaging scaling factor; Due to uncertainties in thermal imaging;

[0347] Determine whether the thermal imaging uncertainty exceeds the corresponding preset gating threshold:

[0348] If the value exceeds the limit, the thermal imaging orientation will be marked as an invalid observation and will not participate in subsequent sequential measurement updates.

[0349] If the observation is not exceeded, the thermal imaging orientation will be marked as a valid observation to facilitate participation in subsequent sequential measurement updates.

[0350] The effective observations and their observation noise covariance corresponding to the acoustic and visual orientations in Example 1, as well as the effective observations and their observation noise covariance corresponding to the thermal imaging orientation, are input into the multi-model state estimation filter.

[0351] In the multi-model state estimation filter, the effective observations and their observation noise covariance corresponding to acoustic azimuth, visual azimuth and thermal imaging azimuth are updated sequentially according to a preset order to output fused azimuth and fused covariance.

[0352] The sequential measurement updates are performed in the preset order of acoustic, visual, and thermal imaging.

[0353] This embodiment introduces thermal imaging perception as a third mode, which can supplement additional orientation observation information in visual degradation scenarios such as low light, strong light, and visual camouflage, thereby further improving the robustness and environmental adaptability of target tracking.

[0354] Specifically, when a visible light camera cannot reliably detect a target due to excessively dark or bright environments, or when the target and background textures are similar, a thermal imaging sensor can still effectively extract the heat source area of ​​the target and output thermal imaging orientation observation by sensing the temperature difference between the target and the background. At the same time, the thermal imaging orientation and the acoustic orientation can complement each other. The acoustic orientation is not affected by illumination, and the thermal imaging orientation is not affected by rotor noise. Together, they make up for the failure risk of the visual modality in degraded scenarios.

[0355] It should be noted that the thermal imaging uncertainty in this embodiment can participate in the exponential scaling and gating judgment of the observation noise covariance according to the same mechanism as acoustic uncertainty and visual uncertainty. This allows the thermal imaging mode to automatically obtain a higher fusion weight when the positioning is reliable, and the weight is reduced or gated and skipped when the positioning is unreliable. In addition, the three modes used in this embodiment can automatically adjust the confidence level allocation of each mode when any mode degenerates, without the need for manual switching or preset fixed weights. This achieves a smooth expansion from two modes to three modes, and significantly improves the stability and safety of UAV target tracking under complex and changing environmental conditions.

[0356] Example 3

[0357] like Figure 5 As shown, this embodiment provides an audio-visual fusion UAV target tracking system based on rotor speed-driven harmonic suppression, including:

[0358] Data acquisition module 301 is used to acquire multimodal data;

[0359] The multimodal data includes multi-channel microphone data, image data of the target to be tracked, and rotor speed data of the UAV itself;

[0360] The mask generation module 302 is used to calculate the blade passing frequency based on the rotor speed data of the UAV and generate a harmonic suppression mask according to the blade passing frequency to suppress the harmonic noise of the UAV rotor.

[0361] The acoustic sensing module 303 is used to calculate the cross power spectrum and signal-to-noise ratio adaptive weight of the multi-channel microphone data, and to perform weighted processing on the cross power spectrum according to the harmonic suppression mask and the signal-to-noise ratio adaptive weight to generate a weighted cross-correlation spectrum to estimate the acoustic orientation and acoustic uncertainty.

[0362] The visual perception module 304 is used to perform target detection on the image data of the target to be tracked, so as to determine the visual orientation of the target to be tracked, and to generate visual uncertainty through a pre-trained evidence regression model.

[0363] The covariance determination module 305 is used to dynamically calculate the acoustic observation noise covariance and the visual observation noise covariance based on the acoustic uncertainty and the visual uncertainty, respectively.

[0364] The gating judgment module 306 is used to determine whether the acoustic uncertainty and / or the visual uncertainty exceed the corresponding preset gating threshold.

[0365] If the number of observations exceeds the limit, the corresponding acoustic and / or visual orientations will be marked as invalid observations and will not be included in subsequent measurement updates.

[0366] If the observation is not exceeded, the corresponding acoustic and / or visual orientation is marked as a valid observation, and the data fusion module 307 is executed.

[0367] The data fusion module 307 is used to perform sequential measurement updates on the effective observations and their corresponding observation noise covariance according to a preset order based on a multi-model state estimation filter, so as to output the fused orientation and fused covariance.

[0368] The UAV control module 308 is used to calculate the fusion confidence based on the fusion covariance, and adaptively scale the control gain according to the fusion orientation and the fusion confidence to generate tracking control commands to control the UAV to track the target.

[0369] In some embodiments, the system further includes a unified messaging interface module 309 and a data recording playback module 310.

[0370] Specifically, the unified message interface module is used to uniformly encapsulate communication messages between various modules;

[0371] The communication messages include observation messages, telemetry messages, command messages, and event messages; each type of communication message includes a timestamp, source identifier, azimuth angle, uncertainty, fusion confidence, flight status, and extended fields.

[0372] Specifically, the data recording and playback module is used to record the observation messages, telemetry messages, command messages and event messages in JSONL format or equivalent sequential log format. During playback, the time axis of the observation data, telemetry data and control commands of each mode is aligned according to the monotonic clock time to reproduce the observation degradation, link loss, state switching and command gating process, which is used for fault tracing and offline parameter tuning.

[0373] Each record contains a version number, system time, and monotonic clock time.

[0374] Example 4

[0375] This embodiment provides a drone, including a body, a flight controller, a processor, a multi-channel microphone array, a camera, a rotor speed acquisition unit, and a memory.

[0376] Specifically, a multi-channel microphone array is deployed at the end of the fuselage arm to collect multi-channel microphone data; a camera is installed on the nose or gimbal of the fuselage to collect target image data; and a rotor speed acquisition unit is connected to the flight controller or electronic speed controller to acquire the rotor speed data of the aircraft.

[0377] Specifically, the memory stores a computer program, and when the processor executes the computer program, it implements the method described in Embodiment 1 or Embodiment 2, generates tracking control commands, and sends the tracking control commands to the flight controller to control the UAV to track the target.

[0378] In this embodiment, the processor is deployed inside the machine body or implemented through an onboard companion computer, and the multi-channel microphone array, the camera, and the rotor speed acquisition unit are all communicatively connected to the processor.

[0379] This embodiment effectively reduces the interference of rotor harmonic noise on acoustic positioning by installing a multi-channel microphone array at the end of the arm, maximizing the array baseline and keeping it away from the propeller noise source. Through the joint deployment of the camera and microphone array, physical spatial complementarity of acoustic and visual perception is achieved.

[0380] Example 5

[0381] This embodiment provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor.

[0382] Specifically, the electronic device is communicatively connected to a multi-channel microphone array, a camera, and a rotor speed acquisition unit to acquire multimodal data; the multimodal data includes multi-channel microphone data, target image data, and rotor speed data of the UAV itself.

[0383] Specifically, when the processor executes the computer program, it implements the method described in Embodiment 1 or Embodiment 2, generates tracking control commands, and sends the tracking control commands to the flight controller through a communication interface to control the UAV to track the target.

[0384] In this embodiment, the electronic device is an airborne companion computer or a ground station device, and the communication interface is a wired communication interface or a wireless communication interface.

[0385] By deploying the perception, fusion, and control algorithms in independent electronic devices, the computing power burden on the main control chip of the UAV flight controller is reduced, which facilitates algorithm upgrades and system maintenance, while also supporting remote monitoring and secure takeover by ground stations.

[0386] Example 6

[0387] This embodiment provides a computer-readable storage medium storing computer-executable instructions for causing a computer to execute a sound-visual fusion UAV target tracking method based on rotor speed-driven harmonic suppression as described in Embodiment 1 or Embodiment 2.

[0388] It should be noted that the computer-readable storage medium in this embodiment may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof.

[0389] More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0390] In this embodiment, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this embodiment, the computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof.

[0391] The computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium, which can send, propagate or transmit a program for use by or in connection with an instruction execution system, apparatus or device.

[0392] Computer programs contained on a computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0393] The computer-readable storage medium described above can be used to write computer programs for executing this embodiment in one or more programming languages ​​or combinations thereof. These programming languages ​​include object-oriented programming languages ​​such as Python, Java, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0394] The above description is merely a preferred embodiment of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects. The scope of the present invention is defined by the appended claims rather than the foregoing description, and thus all variations falling within the meaning and scope of the equivalents of the claims are intended to be included within the present invention.

Claims

1. A method for audio-visual fusion UAV target tracking based on rotor speed-driven harmonic suppression, characterized in that, include: A. Acquire multimodal data; The multimodal data includes multi-channel microphone data, image data of the target to be tracked, and rotor speed data of the UAV itself; B. Calculate the blade passage frequency based on the rotor speed data of the UAV, and generate a harmonic suppression mask according to the blade passage frequency to suppress the harmonic noise of the UAV rotor. C. Calculate the cross-power spectrum and signal-to-noise ratio adaptive weight of the multi-channel microphone data, and perform weighted processing on the cross-power spectrum according to the harmonic suppression mask and the signal-to-noise ratio adaptive weight to generate a weighted cross-correlation spectrum to estimate acoustic orientation and acoustic uncertainty. D. Perform target detection on the image data of the target to be tracked to determine the visual orientation of the target, and generate visual uncertainty through a pre-trained evidence regression model; E. Calculate the acoustic observation noise covariance and the visual observation noise covariance dynamically based on the acoustic uncertainty and the visual uncertainty, respectively; F. Determine whether the acoustic uncertainty and / or the visual uncertainty exceed the corresponding preset gating threshold; If the number of observations exceeds the limit, the corresponding acoustic and / or visual orientations will be marked as invalid observations and will not be included in subsequent measurement updates. If the observation is not exceeded, the corresponding acoustic and / or visual orientation is marked as a valid observation, and step G is executed. G. Based on the multi-model state estimation filter, sequential measurement updates are performed on the effective observations and their corresponding observation noise covariance in a preset order to output the fused orientation and fused covariance. H. Calculate the fusion confidence score based on the fusion covariance, and adaptively scale the control gain according to the fusion orientation and the fusion confidence score to generate tracking control commands to control the UAV to track the target.

2. The method according to claim 1, characterized in that, The function expression for the frequency of the blade passage is: ； In the formula, The frequency at which the blades pass through; This refers to the rotor speed data of the drone itself. This refers to the number of rotor blades; The functional expression for the harmonic suppression mask is: ； In the formula, This is a harmonic suppression mask; H is the highest harmonic order; h is the harmonic order; The soft mask depth is the h-th harmonic. For frequency; denoted as the suppression bandwidth of the h-th harmonic.

3. The method according to claim 1, characterized in that, Step C specifically includes the following steps: The multi-channel microphone data is sampled according to a time window and a fast Fourier transform is performed to obtain the frequency domain representation of each channel; The multi-channel microphone signal is acquired by a multi-channel microphone array deployed on the drone, and the multi-channel microphone signal is a time-domain audio signal. Based on the multi-channel microphone array, several microphone pairs are constructed, and the phase transformation whitening cross power spectrum of each microphone pair is calculated. Based on the frequency domain representation of each channel, the signal power of each frequency point in the frequency domain representation of each channel is calculated, and the noise power of each frequency point is estimated according to the power spectrum low quantile filtering or minimum statistical method. Based on the signal power and noise power of each frequency point, the signal-to-noise ratio (SNR) of each frequency point is calculated, and an adaptive SNR weight is generated according to the SNR. The phase-transformed whitened cross-power spectrum, the harmonic suppression mask, and the signal-to-noise ratio adaptive weights are multiplied point-by-point in the frequency domain and then subjected to inverse Fourier transform to obtain the weighted cross-correlation spectrum. The time delay search domain is determined based on the spacing and sound speed of the microphone pair. Within the time delay search domain, the peak position of the weighted cross-correlation spectrum is used as the estimated time delay, and the estimated time delay mapping is used as the acoustic azimuth angle of the microphone pair. The acoustic azimuth angles of each microphone pair are circularly statistically aggregated, and the weighted sine and cosine sums are calculated to obtain the average synthesized vector length. The complement of the average synthesized vector length is used as the acoustic uncertainty.

4. The method according to claim 3, characterized in that, The functional expression for the phase-transformed whitening cross-power spectrum is: ； In the formula, Phase-transformed whitened cross-power spectrum; For the i-th microphone in the microphone pair; Let be the complex conjugate of the j-th microphone in the microphone pair; The functional expression for the signal-to-noise ratio adaptive weight is: ； In the formula, For signal-to-noise ratio adaptive weights; It is an S-shaped function; The slope coefficient of the sigmoid function; The signal-to-noise ratio at frequency f is expressed in dB. For reference signal-to-noise ratio; The functional expression for the weighted cross-correlation spectrum is: ； In the formula, For weighted cross-correlation spectrum; For time delay variables; For harmonic suppression mask; This is the inverse fast Fourier transform; The functional expression for the acoustic azimuth angle is: ； In the formula, Acoustic azimuth; To estimate the time delay; Speed ​​of sound; The spacing between the microphone pairs; This is a truncation function.

5. The method according to claim 1, characterized in that, Step D specifically includes the following steps: Perform target detection on the image data of the target to be tracked to obtain the coordinates of the detection box and the center point of the detection box; The image data of the target to be tracked is collected by the camera mounted on the drone; The visual azimuth angle is calculated based on the center point of the detection frame and the pre-calibrated camera intrinsic parameters; The target image data is input into a pre-trained evidence regression model to obtain several normal-inverse gamma distribution parameters, and the visual uncertainty is calculated based on the several distribution parameters. The functional expression for the visual azimuth angle is: ； In the formula, Visual azimuth angle; The horizontal pixel coordinates of the center point of the detection box; The x-coordinate of the principal point of the camera; The horizontal focal length of the camera; The functional expression for the visual uncertainty is: ； In the formula, For visual uncertainty; , , All parameters are normal-inverse gamma distribution parameters.

6. The method according to claim 1, characterized in that, Step E specifically includes the following steps: Using the acoustic uncertainty as an exponential scaling factor, the preset acoustic observation noise covariance base value is scaled first, and the first scaling result is truncated with upper and lower limits to obtain the acoustic observation noise covariance. Using the visual uncertainty as an exponential scaling factor, the preset visual observation noise covariance base value is scaled a second time, and the upper and lower limits of the second scaling result are truncated to obtain the visual observation noise covariance. The functional expression for the acoustic observation noise covariance is: ； In the formula, For acoustic observation noise covariance; The base value for the noise covariance of acoustic observations; This is the scaling factor for acoustic uncertainty on the acoustic observation noise covariance; Acoustic uncertainty; The functional expression for the visual observation noise covariance is: ； In the formula, For visual observation noise covariance; The basic value of visual observation noise covariance; This is the scaling factor for visual uncertainty on the visual observation noise covariance; This represents visual uncertainty.

7. The method according to claim 1, characterized in that, Step G specifically includes the following steps: Based on the preset model transition probability matrix, interactive mixing is performed on each sub-model in the multi-model state estimation filter to obtain the mixed initial state and mixed initial covariance of each sub-model. Starting from the initial mixed state and the initial mixed covariance of each sub-model, extended Kalman filter prediction is performed on each sub-model to obtain the prediction state and prediction covariance of each sub-model. Based on the predicted state and predicted covariance of each sub-model, sequential measurement updates are performed on the effective observations to obtain the innovation and innovation covariance of each sub-model, and the updated state estimate and updated covariance of each sub-model are determined according to the innovation and innovation covariance. Based on the innovation and innovation covariance of each sub-model, calculate the likelihood value of each sub-model and update the model probability of each sub-model; The updated state estimate and the updated covariance are weighted and fused according to the model probability to obtain the fused orientation and the fused covariance. When both the acoustic orientation and the visual orientation are marked as invalid observations, the sequential measurement update is not performed. Instead, the predicted state of each sub-model is used as the state estimate of each sub-model, and weighted fusion is performed.

8. The method according to claim 1, characterized in that, Step H specifically includes the following steps: The fusion confidence is calculated based on the fusion covariance, and the azimuth deviation is calculated based on the fusion azimuth. The azimuth deviation is the difference between the target's azimuth and the UAV's current orientation; The PID control gain is adaptively scaled based on the fusion confidence level to obtain the scaled PID control gain. The PID control gain includes proportional gain, integral gain, and derivative gain; When the fusion confidence level is lower than the preset confidence threshold, or when both the acoustic orientation and the visual orientation are marked as invalid observations, the control gain is attenuated or the integral term in the control gain is frozen, and the UAV is controlled to switch to a degraded state or a return-to-home state. When the fusion confidence is not lower than the preset confidence threshold, and the acoustic orientation and the visual orientation are not all marked as invalid observations, the tracking control command is generated based on the orientation deviation and the scaled control gain.

9. A sound-visual fusion UAV target tracking system based on rotor speed-driven harmonic suppression, characterized in that, include: The data acquisition module is used to acquire multimodal data; The multimodal data includes multi-channel microphone data, image data of the target to be tracked, and rotor speed data of the UAV itself; The mask generation module is used to calculate the blade passing frequency based on the rotor speed data of the UAV and generate a harmonic suppression mask according to the blade passing frequency to suppress the harmonic noise of the UAV rotor. The acoustic sensing module is used to calculate the cross power spectrum and signal-to-noise ratio adaptive weight of the multi-channel microphone data, and to perform weighted processing on the cross power spectrum according to the harmonic suppression mask and the signal-to-noise ratio adaptive weight to generate a weighted cross-correlation spectrum to estimate the acoustic orientation and acoustic uncertainty. The visual perception module is used to perform target detection on the image data of the target to be tracked, so as to determine the visual orientation of the target to be tracked, and to generate visual uncertainty through a pre-trained evidence regression model. The covariance determination module is used to dynamically calculate the acoustic observation noise covariance and the visual observation noise covariance based on the acoustic uncertainty and the visual uncertainty, respectively. The gating judgment module is used to determine whether the acoustic uncertainty and / or the visual uncertainty exceed the corresponding preset gating threshold. If the number of observations exceeds the limit, the corresponding acoustic and / or visual orientations will be marked as invalid observations and will not be included in subsequent measurement updates. If the observation is not exceeded, the corresponding acoustic and / or visual orientation is marked as a valid observation, and the data fusion module is executed. The data fusion module is used to perform sequential measurement updates on the effective observations and their corresponding observation noise covariance according to a preset order based on a multi-model state estimation filter, so as to output the fused orientation and fused covariance. The UAV control module is used to calculate the fusion confidence based on the fusion covariance, and adaptively scale the control gain according to the fusion orientation and the fusion confidence to generate tracking control commands to control the UAV to track the target.

10. An electronic device, characterized in that, include: Sensor components are used to acquire multimodal data; The sensor assembly includes a multi-channel microphone array, a camera, and a speed sensor; The processor is connected to the sensor assembly; A memory, connected to the processor, for storing a computer program, wherein the processor executes the computer program to implement the method according to any one of claims 1 to 8; A communication interface is provided for connection to the processor.

Images