Unmanned aerial vehicle sound-light positioning method and device based on multi-modal fusion and manifold consensus

Abstract

The application discloses a kind of unmanned aerial vehicle sound-light positioning method and device based on multi-modal fusion and manifold consensus, comprising: obtaining the audio signal segment of the multi-channel microphone array of unmanned aerial vehicle in current preset time period, and carrying out short-time Fourier transform and multi-channel adaptive spectral subtraction, obtain two-dimensional complex frequency spectrum matrix.This method and device fuse the semantic weight output by deep learning noise reduction model with the physical weight mask calculated, wherein the deep learning noise reduction model suppresses out-of-band random noise, significantly improves the zero-sample target extraction capability in extremely low signal-to-noise ratio and unknown open environment;The wavefront coherence coefficient of each frequency point is calculated for each frequency point screened by the method and device, the spatial response graph is calculated based on the wavefront coherence coefficient, and the global consensus probability graph is constructed based on the spatial response graph, the three-dimensional direction parameter of the target is extracted, and the spatial spectrum needle tip level sharpening is realized.

Description

Technical Field

[0001] This invention belongs to the field of array signal processing and multimodal target detection technology, specifically relating to a UAV acoustic-optical localization method and device based on multimodal fusion and manifold consensus. Background Technology

[0002] With the rapid iteration and popularization of drone technology, low-altitude, slow-speed, small targets, especially various low-altitude, slow-speed, and small drones, have been widely used in many fields such as civilian, public services, and military, playing an important role in scenarios such as agricultural plant protection, logistics distribution, disaster relief, power line inspection, and battlefield reconnaissance.

[0003] For detecting low-altitude, slow-moving, small targets (such as low-altitude, slow-moving, and small UAVs) in complex open environments, existing acoustic positioning arrays and acoustic imaging devices have the following technical shortcomings:

[0004] 1. In complex environments, the acoustic features of a target are easily drowned out by strong wind noise and background ambient sound. Existing single physical rules (such as signal-to-noise ratio thresholds) are easily affected by stable ambient noise, leading to false alarms; while single deep learning blind source separation models have limited zero-sample generalization ability in cross-scene applications, easily resulting in feature omissions. 2. When processing broadband signals from a hundred-channel ultra-large microphone array, due to the limitations of multipath effects and the spatial sampling theorem, traditional broadband controllable response power (SRP) or high-resolution subspace algorithms (such as MUSIC) produce dense spatial sidelobes (i.e., ghost peaks), causing severe jumps in the final positioning trajectory between the real target and the reflected virtual image. 3. Existing audio-visual cameras generally use a coarse linear center projection model for audio-visual fusion, ignoring the nonlinear radial distortion (barrel / pincushion distortion) of wide-angle lenses and the physical installation parallax between the microphone array center and the camera optical center. This results in acceptable target positioning when it is in the center of the image, but once it moves to the edge of the image or the target distance changes drastically, it produces a large angle mapping error, leading to the acoustic cursor failing to catch up with or deviating significantly from the visual target. Summary of the Invention

[0005] The purpose of this invention is to address the problems raised in the background art by proposing a method and device for UAV acoustic-optical localization based on multimodal fusion and manifold consensus.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] This invention proposes a UAV acoustic-optical localization method based on multimodal fusion and manifold consensus, comprising:

[0008] The audio signal segments of the drone's multi-channel microphone array during the current preset time period are obtained, and short-time Fourier transform and multi-channel adaptive spectral subtraction are performed to obtain a two-dimensional complex spectrum matrix.

[0009] The amplitude spectrum of the reference channel in the two-dimensional complex spectrum matrix is ​​extracted, the amplitude spectrum is resampled and the Mel logarithmic amplitude spectrum is calculated. The Mel logarithmic amplitude spectrum is input into the trained deep learning denoising model to obtain the semantic probability mask. The semantic probability mask is averaged along the time dimension, and the averaged result is mapped to the frequency axis consistent with the short-time Fourier transform result through linear interpolation to obtain the semantic weight.

[0010] The physical weight mask is calculated based on the two-dimensional complex spectrum matrix. The semantic weight is fused with the physical weight mask to obtain the fused weight, and K frequency points are selected from the fused weight.

[0011] For the selected K frequency points, the multi-channel microphone array is divided into multiple overlapping sub-arrays. The spatial covariance matrix and wavefront coherence coefficient of each sub-array at the corresponding frequency point are calculated. The spatial response map is calculated based on the wavefront coherence coefficient, and a global consensus probability map is constructed based on the spatial response map. The maximum peak value is found in the global consensus probability map, and weighted centroid interpolation and Kalman filtering are performed on the grid points around the maximum peak value to extract the three-dimensional orientation parameters of the target.

[0012] Based on the target's three-dimensional orientation parameters and camera calibration parameters, the target is projected onto the image plane, that is, the target's three-dimensional orientation parameters are projected into the first two-dimensional pixel coordinates. The theoretical time delay of each channel of the microphone array is calculated using the three-dimensional orientation parameters, and delay summation beamforming is performed. The ambient sound pressure level and signal-to-noise ratio gain before and after beamforming are measured and rendered synchronously with the first two-dimensional pixel coordinates into the video stream acquired by the camera in real time.

[0013] Preferably, the audio signal segments from the multi-channel microphone array of the UAV during the current preset time period are subjected to short-time Fourier transform and multi-channel adaptive spectral subtraction to obtain a two-dimensional complex spectrum matrix, including:

[0014] The audio signal segment is subjected to a short-time Fourier transform using a fixed-length window function. The first preset quantile of the RMS energy is taken as the environmental steady-state noise floor reference, and the second preset quantile of the RMS energy is taken as the effective energy reference. The input signal-to-noise ratio is then calculated.

[0015] The input signal-to-noise ratio is used as the adaptive parameter for multi-channel adaptive spectral subtraction. The over-subtraction coefficient and noise lower limit are calculated respectively. Finally, multi-channel adaptive spectral subtraction is performed to obtain a time-frequency two-dimensional complex spectrum matrix.

[0016] Preferably, the deep learning noise reduction model includes a feature extraction module, an encoder, a bottleneck layer, a decoder, and an output head connected in sequence.

[0017] The feature extraction module includes a depthwise separable convolutional layer, a batch normalization layer, and an activation layer connected in sequence.

[0018] The encoder includes a first downsampling module, a second downsampling module, and a third downsampling module connected in sequence. Each downsampling module includes a max pooling layer, a convolutional layer, and a TF-Cross SSM module connected in sequence. The TF-Cross SSM module includes a time axis scanning branch and a frequency axis scanning branch set in parallel. The input of the TF-Cross SSM module is flattened along the time axis and discretized state space evolution is performed via the time axis scanning branch, and flattened along the frequency axis and discretized state space evolution is performed via the frequency axis scanning branch. The outputs of the time axis scanning branch and the frequency axis scanning branch are respectively normalized by layers and then concatenated into channels. The output of the TF-Cross SSM module is obtained by passing through a gated linear mapping layer.

[0019] The bottleneck layer is a TF-Cross SSM module;

[0020] The decoder includes a first upsampling module, a second upsampling module, and a third upsampling module connected in sequence. Each upsampling module includes a convolutional layer, a subpixel convolution, a skip connection, and a depthwise separable convolution connected in sequence. Specifically, the skip connection in the first upsampling module is a concatenation of the output of the subpixel convolution in the first upsampling module with the output of the second downsampling module; the skip connection in the second upsampling module is a concatenation of the output of the subpixel convolution in the second upsampling module with the output of the first downsampling module; and the skip connection in the third upsampling module is a concatenation of the output of the subpixel convolution in the third upsampling module with the output of the feature extraction module.

[0021] The output header includes a convolutional layer and an activation layer connected in sequence, and the output of the output header is the semantic probability mask.

[0022] Preferably, the step of calculating the physical weight mask based on the two-dimensional complex spectrum matrix, fusing the semantic weights with the physical weight mask to obtain fused weights, and selecting K frequency points from the fused weights includes:

[0023] The average energy sequence and energy variance sequence of the two-dimensional complex spectrum matrix on the time axis are calculated respectively, and the ratio of the energy variance sequence to the average energy sequence is calculated as the transient stability score sequence. The inverse Fourier transform is performed on the two-dimensional complex spectrum matrix in the frequency dimension to calculate the short cepstrum. The cepstrum peak energy corresponding to the preset fundamental frequency range of the UAV rotor in each short cepstrum is extracted to form the harmonic intensity score sequence.

[0024] The transient stability score sequence and harmonic intensity score sequence are normalized respectively, and then the normalized transient stability score sequence and harmonic intensity score sequence are multiplied point by point to obtain the physical weight mask.

[0025] The semantic weights are multiplied point by point with the physical weight mask to obtain the fused weights;

[0026] The fusion weights are sorted in descending order of their numerical values, and the top K frequency points with the largest values ​​are selected.

[0027] Preferably, the process of obtaining the global consensus probability graph includes:

[0028] The multi-channel microphone array is divided into n overlapping subarrays. For the first K selected frequency points, the spatial covariance matrix of each subarray at the current frequency point is calculated. ,right Perform eigenvalue decomposition, extract the largest eigenvalue and the trace of the matrix, and calculate the ratio between the largest eigenvalue and the trace of the matrix as the wavefront coherence coefficient of the current frequency point;

[0029] A generalized cross-correlation phase transformation operation is performed on the spatial covariance matrix to extract pure phase information. The pure phase information is combined with the wavefront coherence coefficient to calculate the spatial response map of each subarray on the three-dimensional spherical grid.

[0030] The median of the response is calculated for each grid point of the spatial response map of the n subarrays to form a global median map. The response variance of the spatial response maps of the n subarrays is calculated for each grid point to form a global variance map. ;

[0031] Calculate the global consensus probability graph using the global median graph and the global variance graph. The calculation formula is as follows:

[0032] ;

[0033] in, This is the preset exponential variance penalty adjustment coefficient.

[0034] Preferably, the step of finding the maximum peak in the global consensus probability graph and performing weighted centroid interpolation and Kalman filtering on the grid points surrounding the maximum peak includes:

[0035] Extract the maximum peak value in the global consensus probability map, take a preset number of grid points around the maximum peak value and perform a weighted centroid interpolation operation to obtain the two-dimensional angle of arrival of the current target, and map the two-dimensional angle of arrival into a coarse three-dimensional spatial direction vector with the physical center of the multi-channel microphone array as the origin.

[0036] The precise three-dimensional spatial direction vector and the first-order rate of change of the current target are used as state variables, and the current coarse three-dimensional spatial direction vector is used as observation variable. These are input into the state update equation of the Kalman filter. Combined with the state variables predicted in the previous time, the current precise three-dimensional spatial direction vector is output, which is the three-dimensional direction parameter of the target.

[0037] Preferably, the three-dimensional orientation parameters of the target are projected into first two-dimensional pixel coordinates. The theoretical time delay of each channel of the microphone array is calculated using the three-dimensional orientation parameters. Delay summation beamforming is performed, and the ambient sound pressure level and signal-to-noise ratio gain before and after beamforming are measured. These are then synchronously rendered to the video stream acquired by the camera in real time, along with the first two-dimensional pixel coordinates.

[0038] Obtain the camera's intrinsic parameter matrix and radial distortion coefficients, and combine them with the displacement vector between the camera and the physical center of the multi-channel microphone array and the camera's currently corrected dynamic extrinsic rotation matrix to map the target's three-dimensional orientation parameters into first two-dimensional pixel coordinates;

[0039] The theoretical time difference of arrival of each microphone in the multi-channel microphone array relative to the physical center of the multi-channel microphone array is calculated using the three-dimensional orientation parameters of the target. Audio signal segments are extracted in the time domain, and delay summation beamforming is performed based on the theoretical time difference of arrival to obtain the target audio signal.

[0040] Calculate the ratio of signal energy to noise energy of the target audio signal to obtain the output signal-to-noise ratio;

[0041] The difference between the output signal-to-noise ratio and the input signal-to-noise ratio is used as the signal-to-noise ratio gain;

[0042] The signal-to-noise ratio gain, output signal-to-noise ratio, and first two-dimensional pixel coordinates are used to form a visualization layer, which is then synchronously overlaid and rendered onto the video stream captured in real time by the camera.

[0043] Preferably, the process of correcting the dynamic extrinsic rotation matrix includes:

[0044] The three-dimensional orientation parameters are projected into second two-dimensional pixel coordinates using the current dynamic extrinsic rotation matrix before correction.

[0045] Extract the video stream of multiple consecutive frames from the camera, and delineate the region of interest with a dynamic radius of R, centered on the second two-dimensional pixel coordinates;

[0046] Within the region of interest, inter-frame differencing and adaptive binarization are performed sequentially, and combined with morphological operations, transient motion bright spots generated by high-frequency rotation of the UAV rotor or movement of the fuselage are captured.

[0047] The moving bright spot is subjected to area and morphological filtering, and connected components are extracted. The area, aspect ratio or contour compactness of each connected component is calculated. Connected components whose area, aspect ratio or contour compactness is within a preset threshold range are selected, and the centroid of the selected connected components is calculated. The centroid is used as the third two-dimensional pixel coordinate.

[0048] Calculate the two-dimensional pixel error between the second and third two-dimensional pixel coordinates. Based on the horizontal and vertical field of view of the camera, calculate the mapping relationship between the two-dimensional pixel error and the spatial offset angle in the dynamic extrinsic rotation matrix before correction.

[0049] Based on the mapping relationship and two-dimensional pixel error, the gradient descent method is used to update the dynamic extrinsic rotation matrix before correction, and a threshold truncation is performed to obtain the corrected dynamic extrinsic rotation matrix.

[0050] Preferably, the loss function of the deep learning denoising model The calculation formula is as follows:

[0051] ;

[0052] in,

[0053] ;

[0054] ;

[0055] ;

[0056] in, For classifying losses, For similarity loss, To compensate for the loss due to continuous harmonic regularization, , and All are weighted parameters. Let be the total number of pixels in the Mel logarithmic amplitude spectrum. and These are the pixel indices for the time axis and the frequency axis, respectively. These are real binary labels manually annotated. The predicted probability output by the deep learning noise reduction model. As a preset value, For along time The forward first-order gradient operation of dimension, For frequency The second-order Laplace operator of dimension , As a category weight balancing factor, The frequency second-order gradient enhancement coefficient, Focusing parameters for difficult samples, The square of the L2 norm.

[0057] A drone acoustic-optical positioning device based on multimodal fusion and manifold consensus includes:

[0058] The data acquisition module is used to acquire audio signal segments from the multi-channel microphone array of the UAV during the current preset time period, and perform short-time Fourier transform and multi-channel adaptive spectral subtraction to obtain a two-dimensional complex spectrum matrix.

[0059] The semantic weight extraction module is used to extract the amplitude spectrum of the reference channel in the two-dimensional complex spectrum matrix, resample the amplitude spectrum and calculate the Mel logarithmic amplitude spectrum, input the Mel logarithmic amplitude spectrum into the trained deep learning denoising model to obtain the semantic probability mask, average the semantic probability mask along the time dimension, and map the average result to the frequency axis consistent with the short-time Fourier transform result through linear interpolation to obtain the semantic weight.

[0060] The multimodal fusion module is used to calculate the physical weight mask based on the two-dimensional complex spectrum matrix, fuse the semantic weights with the physical weight mask to obtain the fused weights, and select K frequency points from the fused weights;

[0061] The manifold consensus positioning module is used to divide the multi-channel microphone array into multiple overlapping sub-arrays for the selected K frequency points, calculate the spatial covariance matrix and wavefront coherence coefficient of each sub-array at the corresponding frequency point, calculate the spatial response map based on the wavefront coherence coefficient, construct a global consensus probability map based on the spatial response map, find the maximum peak value in the global consensus probability map, and perform weighted centroid interpolation and Kalman filtering on the grid points around the maximum peak value to extract the three-dimensional orientation parameters of the target.

[0062] The audio-visual rendering module is used to project the target onto the image plane according to the target's three-dimensional orientation parameters and camera calibration parameters. That is, the target's three-dimensional orientation parameters are projected into the first two-dimensional pixel coordinates. The theoretical time delay of each channel of the microphone array is calculated using the three-dimensional orientation parameters. Delay summation beamforming is performed, and the ambient sound pressure level and signal-to-noise ratio gain before and after beamforming are measured. The results are then rendered synchronously with the first two-dimensional pixel coordinates to the video stream acquired by the camera in real time.

[0063] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0064] 1. The UAV acoustic-optical localization method and device based on multimodal fusion and manifold consensus fuses the semantic weights output by the deep learning denoising model with the calculated physical weight mask. The deep learning denoising model suppresses out-of-band random noise, and the calculation of the physical weight mask removes in-band stationary interference, which significantly improves the zero-sample target extraction capability in extremely low signal-to-noise ratio and unknown open environment.

[0065] 2. This method and apparatus calculates the wavefront coherence coefficient for each selected frequency point, calculates the spatial response map based on the wavefront coherence coefficient, constructs a global consensus probability map based on the spatial response map, finds the maximum peak value in the global consensus probability map, performs weighted centroid and Kalman filtering, extracts the three-dimensional orientation parameters of the target, realizes the tip-level sharpening of the spatial spectrum, and fundamentally solves the problem that causes serious jumps between the real target and the reflected virtual image in the final positioning trajectory;

[0066] 3. This method and device utilize the intrinsic parameter matrix and radial distortion coefficient of the UAV camera, and combine the displacement vector between the camera and the physical center of the multi-channel microphone array and the dynamic extrinsic rotation matrix of the camera after correction, to map the three-dimensional orientation parameters of the target into two-dimensional pixel coordinates. This breaks through the linear mapping bottleneck of traditional acoustic cameras and can ensure pixel-level acoustic-optical fit regardless of whether the target is located at the edge of the image or the depth changes drastically. Attached Figure Description

[0067] Figure 1 This is a flowchart illustrating the UAV acoustic-optical localization method and device based on multimodal fusion and manifold consensus of the present invention.

[0068] Figure 2 This is a schematic diagram of the deep learning noise reduction model of the present invention;

[0069] Figure 3 This is a schematic diagram of the TF-Cross SSM module of the present invention. Detailed Implementation

[0070] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0071] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein in the description of the invention is for the purpose of describing particular embodiments only and is not intended to limit the invention.

[0072] In one embodiment, such as Figures 1-3 As shown, a UAV acoustic-optical localization method based on multimodal fusion and manifold consensus is provided, including:

[0073] Step 1: Obtain audio signal segments from the drone's multi-channel microphone array during the current preset time period, and perform short-time Fourier transform and multi-channel adaptive spectral subtraction to obtain a two-dimensional complex spectrum matrix, specifically:

[0074] The system acquires audio signal segments from the drone's multi-channel microphone array (e.g., 128 channels, but not limited to this, adaptable to the node's hardware computing power) within the current preset time period (e.g., 0.2 seconds). It performs a Short-Time Fourier Transform (STFT, window length set to 2048) on the audio signal segments using a fixed-length window function. It then calculates the RMS energy of each short frame divided by the STFT. The system uses the first preset quantile (e.g., the 20th quantile of the RMS energy as the environmental steady-state noise floor reference NF) and the second preset quantile (e.g., the 75th quantile of the RMS energy as the effective energy reference SR) to calculate the input signal-to-noise ratio (SNR). The calculation formula is: ;

[0075] The input signal-to-noise ratio is used as the adaptive parameter for multi-channel adaptive spectral subtraction. The over-subtraction coefficient and noise lower limit are calculated respectively. Finally, multi-channel adaptive spectral subtraction is performed to obtain a time-frequency two-dimensional complex spectrum matrix.

[0076] Step 2: Extract the amplitude spectrum of the reference channel from the two-dimensional complex spectrum matrix, resample the amplitude spectrum (e.g., resample to 16kHz), and calculate the Mel logarithmic amplitude spectrum. Among them, the frequency surface element dimension Time frame dimension ,aisle Inputting the trained deep learning denoising model yields a semantic probability mask. The semantic probability mask is averaged along the time dimension, and the average result is mapped to a frequency axis consistent with the short-time Fourier transform result (i.e., the result of the short-time Fourier transform in step 1) through linear interpolation to obtain the semantic weights, specifically:

[0077] A deep learning denoising model consists of a feature extraction module, an encoder, a bottleneck layer, a decoder, and an output head connected in sequence.

[0078] The feature extraction module includes sequentially connected depthwise separable convolutions (kernel size 3). 3. Step size 1, padding 1), batch normalization, activation layer (GELU activation function); output dimension of the feature extraction module. Depthwise separable convolution can effectively extract local textures from time-frequency maps while reducing the number of parameters, preventing weak high-frequency harmonics from breaking in the initial stage.

[0079] The encoder includes a first downsampling module (output dimension) connected in sequence. ), second downsampling module (output dimension) ) and the third downsampling module (output dimension) Each downsampling module includes a sequentially connected max-pooling layer (kernel size 2). 2. Stride size of 2), convolutional layer (kernel size of 3) 3. The TF-Cross SSM module (time-frequency dual-axis scanning state space module) with a step size of 1) includes a time axis scanning branch and a frequency axis scanning branch set in parallel. The input of the TF-Cross SSM module is flattened along the time axis and discretized state space evolution is performed through the time axis scanning branch, and flattened along the frequency axis and discretized state space evolution is performed through the frequency axis scanning branch. The outputs of the time axis scanning branch and the frequency axis scanning branch are respectively normalized by the layer and then concatenated into channels. The output of the TF-Cross SSM module is obtained through a gated linear mapping layer (1×1 gated linear mapping (GLU)).

[0080] The equations for the discretized state space evolution of the time-axis scan branch are as follows:

[0081] ;

[0082] ;

[0083] in, Let be the hidden state vector at the current time step. Let be the hidden state vector from the previous time step. The input features for the current timeline scan branch are... This is the output of the timeline scan branch at the current moment. The transition matrix is ​​the state transition matrix. For the input matrix, This is the output matrix;

[0084] The equation for the discretized state space evolution of the frequency axis scanning branch is as follows:

[0085] ;

[0086] ;

[0087] in, This is the hidden state vector of the current frequency channel. Let be the hidden state vector of the previous frequency channel. This is the input feature for the current frequency channel frequency axis scan branch. This is the output of the frequency axis scan branch for the current frequency channel.

[0088] The bottleneck layer is a TF-Cross SSM module, with an output dimension of .

[0089] The decoder includes a first upsampling module (output dimension) connected in sequence. ), second upsampling module (output dimension) ) and the third upsampling module (output dimension) Each upsampling module includes sequentially connected convolutional layers (kernel size 3). 3, with a stride of 1), subpixel convolution, skip connections, and depthwise separable convolution (kernel size of 3). 3, with a step size of 1), wherein the skip connection in the first upsampling module is the concatenation of the subpixel convolution output in the first upsampling module with the output of the second downsampling module (channel concatenation); the skip connection in the second upsampling module is the concatenation of the subpixel convolution output in the second upsampling module with the output of the first downsampling module (channel concatenation); and the skip connection in the third upsampling module is the concatenation of the subpixel convolution output in the third upsampling module with the output of the feature extraction module (channel concatenation).

[0090] The output header consists of sequentially connected convolutional layers (kernel size 1). 1. A step size of 1) and an activation layer (Sigmoid activation function), and the output of the output header is the semantic probability mask ( ).

[0091] It should be noted that the above sequential connections all represent sequential connections from data input to output.

[0092] In the training process of the deep learning noise reduction model, the model is trained and forward inferenced using a training dataset, and the loss function is calculated. The parameters of the deep learning noise reduction model are then updated using backpropagation based on the loss function until the model converges, resulting in a well-trained model. The training dataset is constructed by collecting multi-channel rotor noise data of various types of UAVs under different signal-to-noise ratio environments and simultaneously labeling them with clean single-channel UAV time-frequency signals. In this embodiment, the AdamW optimizer is used during training, with an initial learning rate set to 1 × 10^-4, a batch size of 32, and 100 training iterations. A cosine annealing learning rate decay strategy is also employed to prevent overfitting.

[0093] Loss function of deep learning denoising model The calculation formula is as follows (considering that the weak harmonics of the UAV appear as continuous "thin bright lines" on the time-frequency graph, ordinary pixel-level loss is very likely to cause discontinuities in the prediction mask. Therefore, continuous harmonic regularization penalty loss is introduced. The continuous harmonic regularization penalty loss is used to constrain the continuity of the prediction probability on the time axis and frequency axis to reduce the discontinuity of the semantic probability mask at the thin harmonic structure):

[0094] ;

[0095] in,

[0096] ;

[0097] ;

[0098] ;

[0099] in, For classifying losses, For similarity loss, The loss is penalized for continuous harmonic regularization (by applying second-order difference constraints to the time axis gradient and frequency axis gradient of the predicted probability, penalizing unreasonable breaks and isolated noise). , and All are weighting parameters (preferably 0.4, 0.4, or 0.2). Let be the total number of pixels in the Mel logarithmic amplitude spectrum. and These are the pixel indices for the time axis and the frequency axis, respectively. These are real binary labels manually annotated. The predicted probability output by the deep learning noise reduction model. As a preset value, For along time The forward first-order gradient operation of the dimension (aimed at ensuring short-term continuity). For frequency The second-order Laplace operator of the dimension (aimed at enhancing the comb-like spike structure of harmonics (bright lines)). As a category weight balancing factor, The frequency second-order gradient enhancement coefficient, Focusing parameters for difficult samples, The square of the L2 norm.

[0100] Step 3: Calculate the physical weight mask based on the two-dimensional complex spectrum matrix, fuse the semantic weights with the physical weight mask to obtain the fused weights, and select K frequency points from the fused weights, specifically:

[0101] The average energy sequence and energy variance sequence of the two-dimensional complex spectrum matrix on the time axis are calculated respectively, and the ratio of the energy variance sequence to the average energy sequence is calculated as the transient stability score sequence. The inverse Fourier transform is performed on the two-dimensional complex spectrum matrix in the frequency dimension to calculate the short cepstrum. The peak energy of the cepstrum corresponding to the preset fundamental frequency range of the UAV rotor (where the preset fundamental frequency range is 50Hz ~ 1000Hz) in each short cepstrum is extracted to form the harmonic intensity score sequence.

[0102] The transient stability score sequence and harmonic intensity score sequence are normalized (normalized to the [0,1] interval), and then the normalized transient stability score sequence and harmonic intensity score sequence are multiplied point by point to obtain the physical weight mask.

[0103] The semantic weights and physical weight masks are multiplied point by point to obtain the fusion weights (this point-by-point multiplication fusion is called multimodal fusion).

[0104] The fusion weights are sorted in descending order of their numerical values, and the top K frequency points with the largest values ​​are selected.

[0105] Step 4: For the selected K frequency points, divide the multi-channel microphone array into multiple overlapping sub-arrays, calculate the spatial covariance matrix and wavefront coherence coefficient of each sub-array at the corresponding frequency point, calculate the spatial response map based on the wavefront coherence coefficient, and construct a global consensus probability map based on the spatial response map. Find the maximum peak value in the global consensus probability map, and perform weighted centroid interpolation and Kalman filtering on the grid points around the maximum peak value to extract the three-dimensional orientation parameters of the target (i.e., the UAV) (Step 4 is called the manifold consensus algorithm), specifically:

[0106] The process of obtaining the global consensus probability graph includes:

[0107] The multi-channel microphone array is divided into n overlapping subarrays (specifically, the UAV's 128-channel microphone array is divided into 6 overlapping subarrays, each containing 24 channels). For the selected first K frequency points, the spatial covariance matrix of each subarray at the current frequency point is calculated. ,right Perform eigenvalue decomposition and extract the largest eigenvalue. trace of the matrix The ratio between the largest eigenvalue and the trace of the matrix is ​​calculated as the wavefront coherence coefficient at the current frequency (where the wavefront coherence coefficient is...). The calculation formula is: );

[0108] A generalized cross-correlation phase transformation operation is performed on the spatial covariance matrix to extract pure phase information. The pure phase information is combined with the wavefront coherence coefficient to calculate the spatial response map of each subarray on the three-dimensional spherical grid.

[0109] The median of the response is calculated for each grid point of the spatial response map of the n subarrays to form a global median map. The response variance of the spatial response maps of the n subarrays is calculated for each grid point to form a global variance map. ;

[0110] Calculate the global consensus probability graph using the global median graph and the global variance graph. In this process, the variance term is constructed using an exponential penalty to reduce the impact of discrete sidelobes on the positioning results, thereby enabling the global consensus probability map to effectively eliminate spatial sidelobe interference. The calculation formula is as follows:

[0111] ;

[0112] in, This is the preset exponential variance penalty adjustment coefficient.

[0113] The process involves finding the maximum peak value in the global consensus probability graph and performing weighted centroid interpolation and Kalman filtering on the grid points surrounding the maximum peak value, including:

[0114] Extract the maximum peak value (i.e. the probability with the highest probability value) from the global consensus probability graph, take a preset number of grid points around the maximum peak value and perform a weighted centroid interpolation operation (e.g., take 3×3 neighboring grid points around the maximum peak value and perform a weighted centroid interpolation operation) to obtain the current target's two-dimensional angle of arrival (DOA), and map the target's two-dimensional angle of arrival to the current coarse three-dimensional spatial direction vector with the physical center of the multi-channel microphone array as the origin of the coordinate system.

[0115] A three-dimensional uniform motion model (CV) is constructed. The precise three-dimensional spatial direction vector and the first-order rate of change of the current target are used as state variables, and the current coarse three-dimensional spatial direction vector is used as observation variables. These are input into the state update equation of the Kalman filter. Combined with the state variables predicted in the previous time, the current precise three-dimensional spatial direction vector is output, which is the three-dimensional direction parameter of the target.

[0116] Step 5: Project the target onto the image plane based on the target's three-dimensional orientation parameters and the camera calibration parameters. This involves projecting the target's three-dimensional orientation parameters into first-dimensional pixel coordinates. Using the three-dimensional orientation parameters, calculate the theoretical time delay of each channel in the microphone array. Perform delay summation beamforming, measure the ambient sound pressure level and signal-to-noise ratio gain before and after beamforming, and synchronously render these values ​​along with the first-dimensional pixel coordinates to the video stream acquired by the camera in real time. Specifically:

[0117] Obtain the camera's intrinsic parameter matrix and radial distortion coefficients, and combine them with the displacement vector between the camera and the physical center of the multi-channel microphone array and the camera's currently corrected dynamic extrinsic rotation matrix (which includes yaw and pitch offsets) to map the target's three-dimensional orientation parameters to the first two-dimensional pixel coordinates (completely canceling edge distortion).

[0118] The theoretical time difference of arrival of each microphone in the multi-channel microphone array relative to the physical center of the multi-channel microphone array is calculated by using the three-dimensional orientation parameters of the target. Audio signal segments are extracted in the time domain and delayed summation beamforming (DSB) processing is performed based on the theoretical time difference of arrival to obtain a target audio signal with one-dimensional strong directivity.

[0119] Calculate the ratio of signal energy to noise energy of the target audio signal to obtain the output signal-to-noise ratio;

[0120] The difference between the output signal-to-noise ratio and the input signal-to-noise ratio is used as the signal-to-noise ratio gain;

[0121] The signal-to-noise ratio gain, output signal-to-noise ratio, and first two-dimensional pixel coordinates are used to form a visualization layer, which is then synchronously overlaid and rendered onto the video stream captured by the camera in real time (i.e., synchronously overlaid and rendered onto the RGB real-scene video stream captured by the camera in real time), so that the location of the drone can be seen in the video stream.

[0122] The process of correcting the dynamic extrinsic rotation matrix includes:

[0123] The three-dimensional orientation parameters are projected into second two-dimensional pixel coordinates using the dynamic extrinsic rotation matrix before the current correction. , );

[0124] Extract the video stream of multiple consecutive frames (e.g., three frames) from the camera, and delineate the region of interest with a dynamic radius of R, centered on the second two-dimensional pixel coordinates. The dynamic radius R is adaptively calculated based on the pixel motion span of the UAV target in the previous frame and / or the prediction error of the dynamic extrinsic rotation matrix before the current correction, to ensure that the transient motion bright spot is completely contained within the region of interest.

[0125] Within the region of interest, inter-frame differencing and adaptive binarization are performed sequentially, and combined with morphological operations, transient motion bright spots generated by high-frequency rotation of the UAV rotor or movement of the fuselage are captured.

[0126] The moving bright spot is subjected to area and morphological filtering, and connected components are extracted. The area, aspect ratio, or contour compactness of each connected component is calculated. Connected components whose area, aspect ratio, or contour compactness is within a preset threshold range are selected, and the centroid of the selected connected components is calculated. This centroid is used as the third two-dimensional pixel coordinate. , );

[0127] Calculate the two-dimensional pixel error between the second and third two-dimensional pixel coordinates. , Based on the camera's horizontal and vertical field of view, the mapping relationship between the two-dimensional pixel error and the spatial offset angle in the dynamic extrinsic rotation matrix before correction (i.e., the yaw and pitch offsets in the dynamic extrinsic rotation matrix before correction) is calculated. , );

[0128] Based on the mapping relationship and two-dimensional pixel error, the gradient descent method is used to update the dynamic extrinsic rotation matrix of the camera before correction, and threshold truncation is performed (the updated dynamic extrinsic rotation matrix is ​​truncated to prevent parameter divergence caused by occasional visual noise) to obtain the corrected dynamic extrinsic rotation matrix.

[0129] The update formula for the dynamic extrinsic rotation matrix is ​​as follows:

[0130] ;

[0131] ;

[0132] in, and These are the yaw angle offset and pitch angle offset of the updated dynamic extrinsic rotation matrix, respectively. and These are the yaw angle offset and pitch angle offset of the dynamic extrinsic rotation matrix before correction. It is an adaptive learning rate.

[0133] In a scenario where at least two acoustic array nodes with known relative coordinates work together, each node follows steps 1-4 and outputs the target's three-dimensional orientation parameters with confidence. Then, spatial intersection is performed based on the three-dimensional orientation parameters of each node to obtain the absolute spatial coordinates of the UAV target.

[0134] In another embodiment, another way to obtain semantic weights is as follows:

[0135] The amplitude spectrum of the reference channel in the two-dimensional complex spectrum matrix is ​​extracted, and it is divided into overlapping two-dimensional image blocks, which are then flattened into a one-dimensional sequence.

[0136] For each image patch, position encoding and timestamp embedding are added, and the input is the audio spectrogram transformer (AST) or a conformer network. Specifically, the two-dimensional image patch is used as the input order, and the global long-range dependency features of the time axis and frequency axis are extracted using a multi-head self-attention mechanism. After being mapped by a fully connected layer, the semantic weights are output after attention weighting.

[0137] In another embodiment, another way to extract the three-dimensional orientation parameters of the target is as follows:

[0138] Perform a short-time Fourier transform on the acquired audio signal segments to obtain a multi-channel frequency domain signal matrix. Utilizing fusion weights The multi-channel frequency domain signal matrix is ​​purified at the time-frequency point level to obtain a purified signal matrix. ;

[0139] in accordance with Reconstruct a clean spatial covariance matrix, perform eigenvalue decomposition on the spatial covariance matrix, and divide the eigenvector matrix space into a signal subspace and a noise subspace;

[0140] By utilizing the orthogonality between the array manifold steering vector and the noise subspace, a pseudospectral in the MUSIC space is constructed:

[0141] ;

[0142] in, In azimuth angle Angle of elevation The pseudospectral of the MUSIC space below, In azimuth angle Angle of elevation The array manifold guide vector below, It is the conjugate transpose. For noise subspace;

[0143] The variance penalty (VPSC) mechanism is applied to the MUSIC space pseudospectrum to suppress pseudo-peaks, and the extreme points are searched to obtain the coordinates of the UAV target.

[0144] In another embodiment, in scenarios where visual sensors are completely ineffective, such as at night, in dense fog, or in strong backlight, and where multiple array nodes are working together, another method for obtaining the three-dimensional orientation parameters of the target is as follows:

[0145] Deploy at least two 128-channel acoustic array nodes with known relative GPS / RTK coordinates;

[0146] Each node independently runs the dual-drive mask and manifold consensus algorithm (i.e., steps 2-4) and outputs the target three-dimensional direction vector (DOA ray) with confidence score.

[0147] Based on the DOA rays of multiple nodes, calculate the intersection points or the midpoint of the least common perpendicular line segment of the rays in three-dimensional space, and use them as the global absolute coordinates of the UAV target.

[0148] If a standard test drone (broadcasting its own RTK coordinates) is known to exist in the current airspace, each array node compares its own calculated acoustic positioning point with the received drone's actual coordinates, and reversely calculates the pitch, yaw, and roll errors of its own hardware installation. Through rotation matrix compensation, it achieves pure acoustic closed-loop self-calibration.

[0149] In another embodiment, based on a UAV acoustic-optical localization method based on multimodal fusion and manifold consensus, a UAV acoustic-optical localization device based on multimodal fusion and manifold consensus is also proposed, comprising:

[0150] The data acquisition module is used to acquire audio signal segments from the multi-channel microphone array of the UAV during the current preset time period, and perform short-time Fourier transform and multi-channel adaptive spectral subtraction to obtain a two-dimensional complex spectrum matrix.

[0151] The semantic weight extraction module is used to extract the amplitude spectrum of the reference channel in the two-dimensional complex spectrum matrix, resample the amplitude spectrum and calculate the Mel logarithmic amplitude spectrum, input the Mel logarithmic amplitude spectrum into the trained deep learning denoising model to obtain the semantic probability mask, average the semantic probability mask along the time dimension, and map the average result to the frequency axis consistent with the short-time Fourier transform result through linear interpolation to obtain the semantic weight.

[0152] The multimodal fusion module is used to calculate the physical weight mask based on the two-dimensional complex spectrum matrix, fuse the semantic weights with the physical weight mask to obtain the fused weights, and select K frequency points from the fused weights;

[0153] The manifold consensus localization module is used to divide the multi-channel microphone array into multiple overlapping sub-arrays for the selected K frequency points, calculate the spatial covariance matrix and wavefront coherence coefficient of each sub-array at the corresponding frequency point, calculate the spatial response map based on the wavefront coherence coefficient, construct a global consensus probability map based on the spatial response map, find the maximum peak in the global consensus probability map, and perform weighted centroid interpolation and Kalman filtering on the grid points around the maximum peak to extract the three-dimensional orientation parameters of the target.

[0154] The audio-visual rendering module is used to project the target onto the image plane based on the target's three-dimensional orientation parameters and the camera calibration parameters. Specifically, it projects the target's three-dimensional orientation parameters into the first two-dimensional pixel coordinates, uses the three-dimensional orientation parameters to calculate the theoretical time delay of each channel of the microphone array, performs delay summation beamforming, measures the ambient sound pressure level and signal-to-noise ratio gain before and after beamforming, and renders it synchronously with the first two-dimensional pixel coordinates to the video stream acquired by the camera in real time.

[0155] For specific limitations regarding the UAV acoustic-optical positioning device based on multimodal fusion and manifold consensus, please refer to the limitations of the UAV acoustic-optical positioning method based on multimodal fusion and manifold consensus mentioned above, which will not be repeated here.

[0156] This UAV acoustic-optical localization method and device, based on multimodal fusion and manifold consensus, fuses the semantic weights output by a deep learning denoising model with a calculated physical weight mask. The deep learning denoising model suppresses out-of-band random noise, and the calculation of the physical weight mask removes in-band stationary interference, significantly improving zero-sample target extraction capabilities in extremely low signal-to-noise ratio and unknown open environments. For each selected frequency point, the method and device calculate the wavefront coherence coefficient, calculate the spatial response map based on the wavefront coherence coefficient, and construct a global consensus probability map based on the spatial response map. The method then searches for the maximum peak value in the global consensus probability map. By using weighted centroid and Kalman filtering to extract the target's three-dimensional orientation parameters, the spatial spectrum is sharpened at the tip level, fundamentally solving the problem of severe jumps in the final positioning trajectory between the real target and the reflected virtual image. This method and device utilize the intrinsic parameter matrix and radial distortion coefficient of the UAV's camera, combined with the displacement vector between the physical center of the camera and the multi-channel microphone array and the camera's corrected dynamic extrinsic rotation matrix, to map the target's three-dimensional orientation parameters into two-dimensional pixel coordinates. This breaks through the linear mapping bottleneck of traditional acoustic cameras, ensuring pixel-level acousto-optical alignment regardless of whether the target is located at the edge of the image or there are drastic changes in depth.

[0157] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0158] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0159] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the appended claims.

Claims

1. A UAV acoustic-optical localization method based on multimodal fusion and manifold consensus, characterized in that: The method includes: The audio signal segments of the drone's multi-channel microphone array during the current preset time period are obtained, and short-time Fourier transform and multi-channel adaptive spectral subtraction are performed to obtain a two-dimensional complex spectrum matrix. The amplitude spectrum of the reference channel in the two-dimensional complex spectrum matrix is ​​extracted, the amplitude spectrum is resampled and the Mel logarithmic amplitude spectrum is calculated. The Mel logarithmic amplitude spectrum is input into the trained deep learning denoising model to obtain the semantic probability mask. The semantic probability mask is averaged along the time dimension, and the averaged result is mapped to the frequency axis consistent with the short-time Fourier transform result through linear interpolation to obtain the semantic weight. The physical weight mask is calculated based on the two-dimensional complex spectrum matrix. The semantic weight is fused with the physical weight mask to obtain the fused weight, and K frequency points are selected from the fused weight. For the selected K frequency points, the multi-channel microphone array is divided into multiple overlapping sub-arrays. The spatial covariance matrix and wavefront coherence coefficient of each sub-array at the corresponding frequency point are calculated. The spatial response map is calculated based on the wavefront coherence coefficient, and a global consensus probability map is constructed based on the spatial response map. The maximum peak value is found in the global consensus probability map, and weighted centroid interpolation and Kalman filtering are performed on the grid points around the maximum peak value to extract the three-dimensional orientation parameters of the target. Based on the target's three-dimensional orientation parameters and camera calibration parameters, the target is projected onto the image plane, that is, the target's three-dimensional orientation parameters are projected into the first two-dimensional pixel coordinates. The theoretical time delay of each channel of the microphone array is calculated using the three-dimensional orientation parameters, and delay summation beamforming is performed. The ambient sound pressure level and signal-to-noise ratio gain before and after beamforming are measured and rendered synchronously with the first two-dimensional pixel coordinates into the video stream acquired by the camera in real time.

2. The UAV acoustic-optical localization method based on multimodal fusion and manifold consensus as described in claim 1, characterized in that: The audio signal segments from the multi-channel microphone array of the UAV during the current preset time period are subjected to short-time Fourier transform and multi-channel adaptive spectral subtraction to obtain a two-dimensional complex spectrum matrix, including: The audio signal segment is subjected to a short-time Fourier transform using a fixed-length window function. The first preset quantile of the RMS energy is taken as the environmental steady-state noise floor reference, and the second preset quantile of the RMS energy is taken as the effective energy reference. The input signal-to-noise ratio is then calculated. The input signal-to-noise ratio is used as the adaptive parameter for multi-channel adaptive spectral subtraction. The over-subtraction coefficient and noise lower limit are calculated respectively. Finally, multi-channel adaptive spectral subtraction is performed to obtain a time-frequency two-dimensional complex spectrum matrix.

3. The UAV acoustic-optical localization method based on multimodal fusion and manifold consensus as described in claim 1, characterized in that: The deep learning noise reduction model includes a feature extraction module, an encoder, a bottleneck layer, a decoder, and an output head connected in sequence. The feature extraction module includes a depthwise separable convolutional layer, a batch normalization layer, and an activation layer connected in sequence. The encoder includes a first downsampling module, a second downsampling module, and a third downsampling module connected in sequence. Each downsampling module includes a max pooling layer, a convolutional layer, and a TF-Cross SSM module connected in sequence. The TF-Cross SSM module includes a time axis scanning branch and a frequency axis scanning branch set in parallel. The input of the TF-Cross SSM module is flattened along the time axis and discretized state space evolution is performed via the time axis scanning branch, and flattened along the frequency axis and discretized state space evolution is performed via the frequency axis scanning branch. The outputs of the time axis scanning branch and the frequency axis scanning branch are respectively normalized by layers and then concatenated into channels. The output of the TF-Cross SSM module is obtained by passing through a gated linear mapping layer. The bottleneck layer is a TF-Cross SSM module; The decoder includes a first upsampling module, a second upsampling module, and a third upsampling module connected in sequence. Each upsampling module includes a convolutional layer, a subpixel convolution, a skip connection, and a depthwise separable convolution connected in sequence. Specifically, the skip connection in the first upsampling module is a concatenation of the output of the subpixel convolution in the first upsampling module with the output of the second downsampling module; the skip connection in the second upsampling module is a concatenation of the output of the subpixel convolution in the second upsampling module with the output of the first downsampling module; and the skip connection in the third upsampling module is a concatenation of the output of the subpixel convolution in the third upsampling module with the output of the feature extraction module. The output header includes a convolutional layer and an activation layer connected in sequence, and the output of the output header is the semantic probability mask.

4. The UAV acoustic-optical localization method based on multimodal fusion and manifold consensus as described in claim 1, characterized in that: The physical weight mask calculated based on the two-dimensional complex spectrum matrix is ​​fused with the semantic weights to obtain fused weights, and K frequency points are selected from the fused weights, including: The average energy sequence and energy variance sequence of the two-dimensional complex spectrum matrix on the time axis are calculated respectively, and the ratio of the energy variance sequence to the average energy sequence is calculated as the transient stability score sequence. The inverse Fourier transform is performed on the two-dimensional complex spectrum matrix in the frequency dimension to calculate the short cepstrum. The cepstrum peak energy corresponding to the preset fundamental frequency range of the UAV rotor in each short cepstrum is extracted to form the harmonic intensity score sequence. The transient stability score sequence and harmonic intensity score sequence are normalized respectively, and then the normalized transient stability score sequence and harmonic intensity score sequence are multiplied point by point to obtain the physical weight mask. The semantic weights are multiplied point by point with the physical weight mask to obtain the fusion weights; The fusion weights are sorted in descending order of their numerical values, and the top K frequency points with the largest values ​​are selected.

5. The UAV acoustic-optical localization method based on multimodal fusion and manifold consensus as described in claim 1, characterized in that: The process of obtaining the global consensus probability graph includes: The multi-channel microphone array is divided into n overlapping subarrays. For the first K selected frequency points, the spatial covariance matrix of each subarray at the current frequency point is calculated. ,right Perform eigenvalue decomposition, extract the largest eigenvalue and the trace of the matrix, and calculate the ratio between the largest eigenvalue and the trace of the matrix as the wavefront coherence coefficient of the current frequency point; A generalized cross-correlation phase transformation operation is performed on the spatial covariance matrix to extract pure phase information. The pure phase information is combined with the wavefront coherence coefficient to calculate the spatial response map of each subarray on the three-dimensional spherical grid. The median of the response is calculated for each grid point of the spatial response map of the n subarrays to form a global median map. The response variance of the spatial response maps of the n subarrays is calculated for each grid point to form a global variance map. ; Calculate the global consensus probability graph using the global median graph and the global variance graph. The calculation formula is as follows: ； in, This is the preset exponential variance penalty adjustment coefficient.

6. The UAV acoustic-optical localization method based on multimodal fusion and manifold consensus as described in claim 1, characterized in that: The step of finding the maximum peak in the global consensus probability graph and performing weighted centroid interpolation and Kalman filtering on the grid points surrounding the maximum peak includes: Extract the maximum peak value in the global consensus probability map, take a preset number of grid points around the maximum peak value and perform a weighted centroid interpolation operation to obtain the two-dimensional angle of arrival of the current target, and map the two-dimensional angle of arrival into a coarse three-dimensional spatial direction vector with the physical center of the multi-channel microphone array as the origin. The precise three-dimensional spatial direction vector and the first-order rate of change of the current target are used as state variables, and the current coarse three-dimensional spatial direction vector is used as observation variable. These are input into the state update equation of the Kalman filter. Combined with the state variables predicted in the previous time, the current precise three-dimensional spatial direction vector is output, which is the three-dimensional direction parameter of the target.

7. The UAV acoustic-optical localization method based on multimodal fusion and manifold consensus as described in claim 2, characterized in that: The target's three-dimensional orientation parameters are projected onto the first two-dimensional pixel coordinates. The theoretical time delay of each channel of the microphone array is calculated using the three-dimensional orientation parameters. Delay-summing beamforming is then performed. The ambient sound pressure level and signal-to-noise ratio gain before and after beamforming are measured and synchronously rendered to the video stream captured by the camera in real time, along with the first two-dimensional pixel coordinates. Obtain the camera's intrinsic parameter matrix and radial distortion coefficients, and combine them with the displacement vector between the camera and the physical center of the multi-channel microphone array and the camera's currently corrected dynamic extrinsic rotation matrix to map the target's three-dimensional orientation parameters into first two-dimensional pixel coordinates; The theoretical time difference of arrival of each microphone in the multi-channel microphone array relative to the physical center of the multi-channel microphone array is calculated using the three-dimensional orientation parameters of the target. Audio signal segments are extracted in the time domain, and delay summation beamforming is performed based on the theoretical time difference of arrival to obtain the target audio signal. Calculate the ratio of signal energy to noise energy of the target audio signal to obtain the output signal-to-noise ratio; The difference between the output signal-to-noise ratio and the input signal-to-noise ratio is used as the signal-to-noise ratio gain; The signal-to-noise ratio gain, output signal-to-noise ratio, and first two-dimensional pixel coordinates are used to form a visualization layer, which is then synchronously overlaid and rendered onto the video stream captured in real time by the camera.

8. The UAV acoustic-optical localization method based on multimodal fusion and manifold consensus as described in claim 7, characterized in that: The process of correcting the dynamic extrinsic rotation matrix includes: The three-dimensional orientation parameters are projected into second two-dimensional pixel coordinates using the current dynamic extrinsic rotation matrix before correction. Extract the video stream of multiple consecutive frames from the camera, and delineate the region of interest with a dynamic radius of R, centered on the second two-dimensional pixel coordinates; Within the region of interest, inter-frame differencing and adaptive binarization are performed sequentially, and combined with morphological operations, transient motion bright spots generated by high-frequency rotation of the UAV rotor or movement of the fuselage are captured. The moving bright spot is subjected to area and morphological filtering, and connected components are extracted. The area, aspect ratio or contour compactness of each connected component is calculated. Connected components whose area, aspect ratio or contour compactness is within a preset threshold range are selected, and the centroid of the selected connected components is calculated. The centroid is used as the third two-dimensional pixel coordinate. Calculate the two-dimensional pixel error between the second and third two-dimensional pixel coordinates. Based on the horizontal and vertical field of view of the camera, calculate the mapping relationship between the two-dimensional pixel error and the spatial offset angle in the dynamic extrinsic rotation matrix before correction. Based on the mapping relationship and two-dimensional pixel error, the gradient descent method is used to update the dynamic extrinsic rotation matrix before correction, and a threshold truncation is performed to obtain the corrected dynamic extrinsic rotation matrix.

9. The UAV acoustic-optical localization method based on multimodal fusion and manifold consensus as described in claim 1, characterized in that: The loss function of the deep learning noise reduction model The calculation formula is as follows: ； in, ； ； ； in, For classifying losses, For similarity loss, To compensate for the loss due to continuous harmonic regularization, , and All are weighted parameters. Let be the total number of pixels in the Mel logarithmic amplitude spectrum. and These are the pixel indices for the time axis and the frequency axis, respectively. These are real binary labels manually annotated. The predicted probability output by the deep learning noise reduction model. As a preset value, For along time The forward first-order gradient operation of dimension, For frequency The second-order Laplace operator of dimension , As a category weight balancing factor, The frequency second-order gradient enhancement coefficient, Focusing parameters for difficult samples, The square of the L2 norm.

10. A drone acoustic-optical positioning device based on multimodal fusion and manifold consensus, characterized in that, include: The data acquisition module is used to acquire audio signal segments from the multi-channel microphone array of the UAV during the current preset time period, and perform short-time Fourier transform and multi-channel adaptive spectral subtraction to obtain a two-dimensional complex spectrum matrix. The semantic weight extraction module is used to extract the amplitude spectrum of the reference channel in the two-dimensional complex spectrum matrix, resample the amplitude spectrum and calculate the Mel logarithmic amplitude spectrum, input the Mel logarithmic amplitude spectrum into the trained deep learning denoising model to obtain the semantic probability mask, average the semantic probability mask along the time dimension, and map the average result to the frequency axis consistent with the short-time Fourier transform result through linear interpolation to obtain the semantic weight. The multimodal fusion module is used to calculate the physical weight mask based on the two-dimensional complex spectrum matrix, fuse the semantic weights with the physical weight mask to obtain the fused weights, and select K frequency points from the fused weights; The manifold consensus positioning module is used to divide the multi-channel microphone array into multiple overlapping sub-arrays for the selected K frequency points, calculate the spatial covariance matrix and wavefront coherence coefficient of each sub-array at the corresponding frequency point, calculate the spatial response map based on the wavefront coherence coefficient, construct a global consensus probability map based on the spatial response map, find the maximum peak value in the global consensus probability map, and perform weighted centroid interpolation and Kalman filtering on the grid points around the maximum peak value to extract the three-dimensional orientation parameters of the target. The audio-visual rendering module is used to project the target onto the image plane according to the target's three-dimensional orientation parameters and camera calibration parameters. That is, the target's three-dimensional orientation parameters are projected into the first two-dimensional pixel coordinates. The theoretical time delay of each channel of the microphone array is calculated using the three-dimensional orientation parameters. Delay summation beamforming is performed, and the ambient sound pressure level and signal-to-noise ratio gain before and after beamforming are measured. The results are then rendered synchronously with the first two-dimensional pixel coordinates to the video stream acquired by the camera in real time.

Images