Audio-visual fusion unmanned aerial vehicle detection and positioning method

Abstract

The invention discloses an audio-visual fusion unmanned aerial vehicle detection and positioning method, and belongs to the technical field of unmanned aerial vehicle detection and positioning, and the method comprises the steps: collecting sound signals of a detection region through a sound array, obtaining the local strongest direction of the sound signals, and forming an unmanned aerial vehicle candidate direction set; signals of candidate orientations of all unmanned aerial vehicles are directionally received, whether the unmanned aerial vehicles exist in the candidate orientations or not is judged through the unmanned aerial vehicle sound signal anomaly detector based on deep learning and the unmanned aerial vehicle image target detector based on deep learning, and an unmanned aerial vehicle orientation set is obtained; acquiring an image of a detection area through a binocular camera to obtain a depth image of a reference camera visual angle; uniquely corresponding the orientation of the unmanned aerial vehicle to the determined position of the depth image, and taking the depth value of the position as the distance of the unmanned aerial vehicle; and combining the unmanned aerial vehicle azimuth and the unmanned aerial vehicle distance to obtain three-dimensional coordinates of the unmanned aerial vehicle. The unmanned aerial vehicle can be identified and positioned by adopting an audio-visual fusion method.

Description

Technical Field

[0001] This invention belongs to the field of unmanned aerial vehicle (UAV) detection and positioning technology, and more specifically, relates to an audiovisual fusion UAV detection and positioning method. Background Technology

[0002] The low-altitude economy is a strategic emerging industry in my country, and drones are an important carrier within it, playing a crucial role in public services such as express delivery and food delivery, as well as in professional fields such as agricultural and forestry seeding and fire rescue. In addition, amateur enthusiasts use drones for aerial photography and other activities. However, with the increase in low-altitude drones, incidents such as flight path conflicts and uncontrolled collisions with buildings and facilities occur frequently, posing challenges to the orderly development of the low-altitude economy and the safe supervision of drones.

[0003] Unmanned aerial vehicle (UAV) detection and localization technologies mainly include visual detection, voiceprint recognition, and radio frequency detection methods. For example, in visual detection, invention CN120182322A discloses a computer vision-based method for tracking the nonlinear motion of small UAV targets, and invention CN120279511A discloses a YOLOv11-based method for UAV identification and localization in airport environments; however, neither has achieved spatial distance localization. In voiceprint recognition, invention CN118969000A discloses a UAV identification method using a complex convolutional recurrent neural network and voiceprint features, but it cannot distinguish UAVs from other interfering sound sources. Regarding multi-feature fusion, invention CN112198476A discloses a three-dimensional sound source localization method based on binocular vision and beamforming. It first uses a binocular camera for visual three-dimensional reconstruction, then uses an acoustic array for sound pressure scanning, selecting the point with the strongest sound pressure as the sound source location, but it cannot identify the target type. Invention CN120263948A discloses a UAV localization method based on images and infrasound, characterized by independent visual and auditory localization and weighted results, but it requires the UAV to carry an infrasound transmission module, and the poor directionality of infrasound leads to poor localization performance. Invention CN120319264A discloses a UAV classification method combining audio and radio frequency, but it only utilizes monophonic data and cannot remove spatial sound field interference. Invention CN120257058A discloses a UAV identification method based on a multimodal large model and modal data such as radar, optical, and radio, but it suffers from high deployment difficulty and other engineering problems.

[0004] Based on a survey of existing drone identification and positioning technologies, it was found that existing drone detection technologies and devices have problems such as difficulty in flexible deployment, potential interference and damage to drones, and limitations imposed by radio regulations, which restrict their application scope and promotion. Summary of the Invention

[0005] In view of the above-mentioned defects or improvement needs of existing technologies, the present invention provides an audiovisual fusion drone detection and positioning method, thereby solving the technical problems of poor positioning performance and high deployment difficulty of existing drone detection and positioning methods.

[0006] To achieve the above objectives, according to one aspect of the present invention, an audiovisual fusion unmanned aerial vehicle (UAV) detection and positioning method is provided, comprising: S1, receive acoustic signals from the detection area through an acoustic array, construct a spatial power spectrum based on the acoustic signals, and search for the location in the detection area where the acoustic signals are strongest locally in the spatial power spectrum, as a candidate location set for the UAV; S2, acquire image blocks of each candidate drone location through a binocular camera, and determine whether there is a drone in each candidate drone location based on the drone sound signal anomaly detector and the drone image target detector. When the output of both the drone sound signal anomaly detector and the drone image target detector is "yes", it is determined that there is a drone in the candidate drone location. All candidate drone locations with drones constitute the drone location set. S3, predict the disparity of each point in the reference camera image in the binocular camera using a binocular disparity predictor, and convert the disparity value into a depth value using the baseline distance and focal length obtained by the binocular camera calibration in advance, to obtain a depth image from the perspective of the reference camera. S4. Based on the center alignment relationship between the acoustic array and the reference camera, the azimuth coordinates of the UAV are mapped to the corresponding position in the depth image. At this time, the depth value at that position is used as the vertical distance of the UAV. S5 establishes three-dimensional coordinates based on the drone's orientation and vertical distance, thereby obtaining the drone's three-dimensional coordinates and enabling drone identification and spatial positioning.

[0007] Furthermore, the audiovisual fusion drone detection and localization method of the present invention is implemented through an audiovisual fusion drone detection and localization device, which includes an acoustic sensor array (i.e., an acoustic array), two cameras, and a storage and computing unit. One camera is located at the center of the acoustic array as a reference camera, i.e., the first camera, and the other camera is the second camera; the two cameras are placed horizontally.

[0008] Furthermore, prior to steps S1 to S5, an offline component is also included, specifically comprising: Calibrate all cameras to obtain the camera intrinsic parameter matrix. Camera baseline distance and epipolar correction parameters. Camera intrinsic parameter matrix. for

[0009] in, It is the focal length of the camera lens. and These are the width and height of the camera's image sensor, respectively. and These are the x and y coordinates of the center of the camera's image sensor in the camera's output image, respectively.

[0010] Preferably, a checkerboard calibration plate is used to calibrate the binocular camera.

[0011] Furthermore, the offline component of the audiovisual fusion drone detection and localization algorithm trains a deep learning-based drone acoustic signal anomaly detector. This anomaly detector can specifically distinguish between acoustic signals caused by drones and other acoustic signals.

[0012] Preferably, the anomaly detector has an encoder-decoder structure, aiming to reconstruct the spectrum of the UAV acoustic signal with minimal mean square error. For any test acoustic signal, if the spectrum of the test acoustic signal can be reconstructed with a mean square error lower than a preset threshold... If the mean square error is reconstructed, then the test acoustic signal comes from the UAV; otherwise, it is not generated by the UAV.

[0013] Furthermore, the offline component of the audiovisual fusion drone detection and localization algorithm trains a deep learning-based drone image target detector. This target detector can identify whether any image patch contains a drone.

[0014] Preferably, the target detector is an image classifier that obtains a classification result; or an anomaly detector that obtains a reconstructed image patch. For any test image patch, based on the classification result or whether the reconstruction error is lower than a preset threshold... The system determines whether a drone is included. The target detector has the capability to identify small targets.

[0015] Furthermore, the offline part of the audiovisual fusion UAV detection and localization algorithm trains a deep learning-based binocular disparity predictor. The binocular disparity predictor predicts the disparity values ​​of each point in the reference camera image based on the binocular images. Preferably, the deep learning-based binocular disparity predictor includes four steps: binocular image feature extraction, cost volume construction, cost aggregation, and disparity regression. The cost volume construction, cost aggregation, and disparity regression follow the GwcNet method. Binocular image feature extraction employs a neural network based on local and global feature extractors. The local feature extractor uses a ResNet-34 structure, but excludes the fully connected layers used for image classification; the global feature extractor uses a U-Net-like structure. The global feature extractor for binocular image feature extraction includes improved shrinking and expanding paths. The shrinking path is characterized by using a pixel unshuffle operation to gradually reduce the spatial dimension of the features. The expanding path is characterized by using a pixel shuffle operation to gradually increase the spatial dimension of the features.

[0016] Preferably, step S1 specifically includes the following steps: S11, receiving acoustic signals from the detection area via an acoustic array, wherein the acoustic array includes Each microphone, all microphones with Sampling frequency synchronous sampling Next, obtain the temporal snapshot data matrix. ,matrix Include lines and List real numbers, that is, numbers with the shape of and ;in, Represents the set of real numbers; S12, for the matrix Perform a Fast Fourier Transform on each row and extract the positive frequency components to obtain the frequency domain snapshot data matrix. ,matrix The elements belong to the set of complex numbers. ,Right now ;in, ;matrix The OK( ), No. List( The element of ) represents the first element. In the signal received by each microphone, the frequency The amplitude and phase; where , Indicates rounding down; S13, Select center frequency That is, selecting a matrix The The column yields the frequency domain snapshot data vector. ,and ; S14, Select Each acoustic scanning point has a line connecting it to the center of the acoustic array, representing the direction of the incoming acoustic signal. The main lobe of the beamformer is adjusted sequentially to scan all the points. The direction of the incoming wave is used to obtain the beamformer. Each response outputs a value, then the power is calculated; all Each power constitutes a spatial power spectrum; S15, Search for local maxima on the spatial power spectrum. The values ​​of the local maxima should be higher than a preset threshold. The incoming wave directions of all local maxima corresponding to the scanning points constitute the candidate azimuth set of the UAV.

[0017] Preferably, the beamformer in step S14 is in the... The response output in each direction is

[0018] in, It is the beamformer at the frequency , No. Weight vectors in each direction.

[0019] Further, in step S14, a delay-sum (DAS) beamformer is selected, and far-field conditions are applied. At this point, The expression is

[0020] in, It is the first Plane waves in all directions at frequencies wave number at that point It is the first The coordinates of each array element.

[0021] Preferably, step S2 specifically includes the following steps: S21, for each candidate azimuth of the UAV, the main lobe of the beamformer is adjusted to that direction, and the beamformer is required to suppress the signals of all other candidate azimuths of the UAV, so as to sequentially receive the acoustic signals of each candidate azimuth of the UAV and obtain the spectrum. S22, For each candidate drone location, a deep learning-based drone acoustic signal anomaly detector is used to determine whether the extracted acoustic signal spectrum is caused by a drone. S23, for each candidate drone location, select an image patch on the candidate region image of the reference camera, and use a deep learning-based drone image target detector to determine whether the extracted image patch contains a drone; S24, all candidate drone locations that were determined to be "yes" in both steps S22 and S23 are added to the drone location set. The drone location is represented by coordinates. Indicates; among which, It is the angle between the line connecting the drone and the center of the acoustic array and the ground. It is the projection of the line connecting the drone and the center of the acoustic array onto the ground, and the ground-preset... The angle between directions.

[0022] Preferably, step S21 selects a minimum variance distortion-free response beamformer, i.e., an MVDR beamformer, which is used in the candidate azimuth of the UAV. and frequency The weight vector at that point is

[0023] in, It is the noise covariance matrix. The acoustic array is related to the candidate azimuth of the drone. of, in frequency The manifold vector on the surface. The MVDR beamformer operates in a narrowband, and the above formula needs to be calculated for the entire narrowband.

[0024] Preferably, in step S23, the candidate azimuth of the UAV... The coordinates of the center of the selected image patch on the candidate region image of the reference camera. The relationship is

[0025] in, , , and All data are derived from the camera intrinsic parameter matrix obtained from camera calibration. .

[0026] Preferably, step S3 specifically includes the following steps: S31, images of the detection area are acquired by a binocular camera, and the epipolar correction parameters obtained from camera calibration are used to perform epipolar correction on the two images, so that the object points in the real world are in the same row on the two images. S32, based on two epipolar-corrected images, uses a deep learning-based binocular disparity predictor to predict the disparity values ​​of each point in the reference camera image; S33 uses the baseline distance and focal length obtained from camera calibration to convert the parallax value into a depth value:

[0027] in, and These are the x-coordinates of the image points of the real-world object points as perceived by the reference camera and the second camera, respectively. This is parallax; It is the depth value of the object point from the perspective of the reference camera. It is the straight-line distance between the optical centers of the reference camera and the second camera, i.e., the baseline distance; It refers to the focal length of the two cameras.

[0028] Preferably, step S4 is based on the formula Positioning of the drone A unique location corresponding to a depth image ,in, , , and All data are derived from the camera intrinsic parameter matrix obtained from camera calibration. .

[0029] Preferably, the three-dimensional coordinates of the UAV for ,in Location of the drone. The vertical distance for the drone.

[0030] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: 1. The audiovisual fusion method for UAV detection and localization proposed in this invention uses an audiovisual fusion approach to identify and locate UAVs. An acoustic array responsible for hearing and a binocular camera responsible for vision respectively acquire acoustic and image signals. The UAV is identified using an acoustic signal anomaly detector and an image target detector. A spatial power spectrum is constructed based on the acoustic signals, and the location with the strongest local acoustic signal in the detection area is searched within the spatial power spectrum as a candidate location for the UAV. A binocular disparity predictor predicts the disparity of each point in the reference camera image in the binocular camera. The disparity values ​​are converted into depth values ​​using the baseline distance and focal length obtained from the pre-calibrated binocular camera, resulting in a depth image from the reference camera's perspective. The UAV's azimuth coordinates are mapped to the corresponding position in the depth image, and the depth value at that position is taken as the UAV's vertical distance. This invention achieves precise spatial positioning by separately measuring the UAV's azimuth and vertical distance.

[0031] 2. The audiovisual fusion drone detection and positioning method proposed in this invention uses auditory features responsible for drone identification, which are present in all drones, and visual features responsible for drone ranging, which are not limited to specific object types. This makes the technical solution of this invention applicable to a variety of drones and has universality.

[0032] 3. The audiovisual fusion UAV detection and positioning method proposed in this invention uses visual and auditory sensing methods that do not transmit signals externally, making it a purely passive solution with minimal interference to the external environment; the sensor components are simple and readily available, and the device is easy to manufacture and deploy. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the audiovisual fusion drone detection and positioning system according to an embodiment of the present invention; Figure 2 This is a schematic diagram of a binocular disparity predictor based on deep learning according to an embodiment of the present invention; Figure 3 This is a flowchart of the online part of the audiovisual fusion UAV detection and positioning algorithm in an embodiment of the present invention; Figure 4 This is a schematic diagram of the acoustic array and binocular camera used in the embodiments of the present invention; Figure 5This is a neural network structure diagram of the binocular disparity predictor based on deep learning used in the embodiments of the present invention; Figure 6 This is a flowchart illustrating step S1 of an embodiment of the present invention; Figure 7 This is a flowchart illustrating step S2 of an embodiment of the present invention; Figure 8 This is a flowchart illustrating step S3 of an embodiment of the present invention.

[0034] In all the accompanying drawings, the same reference numerals are used to denote the same elements or structures, including: 1, acoustic sensor array; 2, binocular camera; 3, detection area; 4, acoustic scanning point. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0036] Example 1 like Figure 1 As shown, one embodiment of the present invention provides an audiovisual fusion drone detection and positioning system. The system includes one acoustic sensor array (i.e., an acoustic array) and two cameras (i.e., binocular cameras), and also includes a storage and computing unit. Figure 4 As shown, the acoustic array and camera selected in this embodiment have the following characteristics: (1) The acoustic array contains 64 acoustic sensors; all acoustic sensors are on the same plane, which is referred to as the array plane in this embodiment. All acoustic sensors are arranged in 8 spiral arms; looking along the direction in which the acoustic sensors receive sound, i.e., the detection direction of the acoustic array, the spiral arms are numbered A~H in a counterclockwise direction. Each spiral arm has 8 acoustic sensors, numbered 1~8 sequentially from the center to the edge of the acoustic array, and connected by dashed lines. To make Figure 4 The array is clearly readable, and element 1 of each spiral arm is not numbered. The distances of acoustic sensors 1 and 8 on each spiral arm from the array center are approximately 0.009 meters and 0.075 meters, respectively. Acoustic sensors with the same numerical designation on adjacent spiral arms, such as A1 and B1, form a 45° angle with the line connecting them to the array center. Adjacent acoustic sensors on the same spiral arm, such as A1 and A2, form a 30° angle with the line connecting them to the array center. A coordinate axis is selected on the array plane. and Let the coordinate axes Pointing to the detection area of ​​the acoustic array. The three coordinate axes are mutually perpendicular, and their outer product satisfies...

[0037] Define coordinate axes The direction is from left to right, and the coordinate axes are... The direction is from top to bottom, and the coordinate axes are... The direction is from back to front. The upper limit of the response frequency of each acoustic sensor is not less than 75 kHz; (2) The line connecting the optical centers of the two cameras and The axis is in the same direction, and the observation direction is the same as... The axes are aligned. The baseline distance between the two cameras. The distance is 0.15 meters. Each camera has a resolution of 1920×1080 and a focal length of 8 mm. The reference camera (i.e., the first camera) is located at the center of the acoustic array, and the second camera is located outside the acoustic array. (3) The above-mentioned acoustic array and camera also have power supply and data interfaces and cables.

[0038] Example 2 This invention provides a method for detecting and locating audiovisual fusion drones, which consists of an offline part and an online part.

[0039] Specifically, the offline part of the audiovisual fusion drone detection and localization algorithm includes calibrating a binocular camera, training a deep learning-based drone acoustic signal anomaly detector, training a deep learning-based drone image target detector, and training a deep learning-based binocular disparity predictor.

[0040] To further explain, this embodiment uses a checkerboard calibration board with alternating black and white squares to measure the camera's intrinsic parameter matrix, the stereo camera's baseline distance, and epipolar correction parameters. The calibration board has 12 squares in width and 9 squares in height. First, at least 40 sets of images are captured using the stereo camera, ensuring the checkerboard is clearly and completely visible in each set of two images. The calibration board should appear in the four corners, four boundaries, and center of the camera images, and should be rotated along the long side, short side, and the central axis perpendicular to the checkerboard pattern, thus facing the camera in different orientations. Then, this embodiment uses subroutines provided by OpenCV to calculate the camera's intrinsic parameter matrix, camera baseline distance, and epipolar correction parameters.

[0041] To further explain, this embodiment employs a deep learning-based drone acoustic signal anomaly detector to identify drones. The input feature used for drone identification is the spectrum of the drone's acoustic signal. The neural network has an encoder-decoder structure, where the encoder compresses the input features into a latent space, and the decoder recovers the features from the latent space to the input features using the minimum mean square error criterion. For any given test sample, if the spectrum of the sample can be recovered by the neural network with a small mean square error, then the acoustic signal of the sample is considered to originate from a drone; otherwise, it does not originate from a drone.

[0042] Furthermore, this embodiment employs a deep learning-based drone image target detector to achieve drone identification, forming a dual drone identification system based on both sound and image. The input to the target detector is an original image patch from the first camera. The target detector can be a classifier that outputs an image category, with the classification result indicating whether the original image patch contains a drone. Alternatively, the target detector can be an anomaly detector that outputs a reconstructed image patch; if the reconstruction error is small, the original image patch is considered to contain a drone. The target detector also possesses small target recognition capabilities.

[0043] To further explain, this embodiment uses a deep learning-based binocular parallax predictor to measure the distance to the drone. For example... Figure 2 As shown, the binocular disparity predictor includes four steps: binocular image feature extraction, cost volume construction, cost aggregation, and disparity regression. The cost volume construction, cost aggregation, and disparity regression steps follow the methods of GwcNet.

[0044] like Figure 5 As shown, the stereo image feature extraction in this embodiment employs a feature extractor consisting of a local feature extractor and a global feature extractor. The local feature extractor uses a ResNet-34 structure, but excludes the last layer, i.e., the fully connected layer used for image classification. The global feature extractor adopts a U-Net-like structure, including improved shrinking and expanding paths. The shrinking path is characterized by the use of a pixel unshuffle operation, which reduces the spatial size of the feature map and increases the number of channels without reducing the original features. The expanding path is characterized by the use of a pixel shuffle operation, thereby gradually increasing the spatial dimension of the features; where pixel shuffle is the inverse operation of pixel unshuffle. The feature maps obtained by the global and local feature extractors are concatenated along the channel dimension.

[0045] like Figure 5 As shown, the global feature extractor in this embodiment also employs a channel attention-based Transformer, which can reduce the computational cost of Transformer attention operations.

[0046] Specifically, such as Figure 3 As shown, the online portion of the audiovisual fusion drone detection and localization method includes the following steps: S1, receive acoustic signals from the detection area through an acoustic array, construct a spatial power spectrum based on the acoustic signals, and search for the location in the detection area where the acoustic signals are strongest locally in the spatial power spectrum, as a candidate location set for the UAV; S2, acquire image blocks of each candidate drone location through a binocular camera, and determine whether there is a drone in each candidate drone location based on the drone sound signal anomaly detector and the drone image target detector. When the output of both the drone sound signal anomaly detector and the drone image target detector is "yes", it is determined that there is a drone in the candidate drone location. All candidate drone locations with drones constitute the drone location set. S3, predict the disparity of each point in the reference camera image in the binocular camera using a binocular disparity predictor, and convert the disparity value into a depth value using the baseline distance and focal length obtained by the binocular camera calibration in advance, to obtain a depth image from the perspective of the reference camera. S4. Based on the center alignment relationship between the acoustic array and the reference camera, the azimuth coordinates of the UAV are mapped to the corresponding position in the depth image. At this time, the depth value at that position is used as the vertical distance of the UAV. S5 establishes three-dimensional coordinates based on the drone's orientation and vertical distance, thereby obtaining the drone's three-dimensional coordinates and enabling drone identification and spatial positioning.

[0047] Specifically, such as Figure 6 As shown, step S1 includes the following sub-steps: S11. Receive acoustic signals from the detection area via an acoustic array. The acoustic array's... A microphone Sampling frequency synchronous sampling Next, obtain the temporal snapshot data matrix. .

[0048] When selecting the number of simultaneous samplings, i.e., the number of time-domain snapshots, the following antagonistic factors can be considered: (1) A higher number of time-domain snapshots can improve frequency resolution and obtain richer frequency information, which is beneficial for subsequent UAV identification based on anomaly detection. (2) A higher number of time-domain snapshots will increase the computational pressure on the system and prolong the time to acquire a time-domain snapshot data matrix, thus reducing the real-time performance of the system response.

[0049] S12, for the matrix Perform a Fast Fourier Transform on each row and extract the positive frequency components to obtain the frequency domain snapshot data matrix. ,Right now ; S13. Select center frequency That is, selecting a matrix The The column yields the frequency domain snapshot data vector. .

[0050] The selection of the imaging center frequency can take into account the following antagonistic factors: (1) As the center frequency increases, better spatial resolution can generally be obtained, thereby more accurately locating the candidate direction of the UAV, which is beneficial for subsequent directional signal reception and UAV identification. (2) As the center frequency increases, the attenuation rate of high-frequency acoustic signals with the propagation distance increases, thus limiting the maximum distance at which the system can identify and locate UAVs.

[0051] S14, Select There are several acoustic scanning points, and the direction of the line connecting each scanning point to the center of the acoustic array represents the direction of arrival of the acoustic signal. The main lobe of the beamformer is adjusted sequentially to scan all... The direction of the incoming wave is used to obtain the beamformer. Each response outputs a power. Each power constitutes a spatial power spectrum.

[0052] This embodiment provides two methods for selecting acoustic scanning points: (1) In some optional implementations, the acoustic scanning point is selected in a rectangular coordinate system: first, the distance from the optical center of the camera is selected as the focal length. In the pinhole imaging model, the width and height of the imaging plane are the same as the width and height of the camera's photosensitive element; then, the number of grid points to be selected at equal intervals along the width and height directions is determined. and Finally, the grid is divided at equal intervals. For those located in The acoustic scanning point, and the direction of the incoming wave it corresponds to. for

[0053] (2) In some alternative implementations, the acoustic scanning point is selected in spherical coordinates: radius The polar angle can be chosen arbitrarily, but is generally taken as 1 for ease of calculation; (Also known as pitch angle) is the angle between the line connecting the acoustic scanning point and the center of the acoustic array and the array plane, and its value range is... azimuth It is the angle between the projection of the line connecting the acoustic scanning point and the center of the acoustic array onto the array plane and the left-right direction of the array, and its value range is... Select at equal intervals along the polar angle and azimuth angle dimensions respectively. and Scan points; finally, divide the grid at equal intervals, at this point... And the direction of the incoming wave at the scanning point is precisely... No additional calculations are required.

[0054] Generally speaking, at frequency , No. In each direction, given the weight vector of the beamformer The output of the beamformer is

[0055] Given a fixed acoustic scanning point and imaging center frequency, all required weight vectors can be calculated offline. This embodiment employs a delay-summation (DAS) beamformer for scanning and obtains the response output, applying the far-field condition that the acoustic signals arriving at each acoustic sensor from the same direction have exactly one phase difference. At frequency... , No. In each direction, the weight vector of the DAS beamformer for

[0056] S15. Search for local maxima on the spatial power spectrum. The values ​​of the local maxima should be higher than a preset threshold. The incoming wave directions of all local maxima corresponding to the scan points constitute the candidate azimuth set of the UAV.

[0057] In this embodiment, the window size for searching local maxima is set to 3×3. For acoustic scanning points selected in Cartesian or spherical coordinate systems, this means that the deviation between each scanning point within the window and the center scanning point in both directions—whether along the width and height directions or along the polar and azimuth directions—does not exceed one grid spacing. Threshold This is the mean of the spatial power spectrum.

[0058] Specifically, such as Figure 7 As shown, step S2 includes the following sub-steps: S21. For each candidate direction of the UAV, adjust the main lobe of the beamformer to that direction, and require the beamformer to suppress the signals of all other candidate directions of the UAV, so as to sequentially receive the acoustic signals of each candidate direction of the UAV and obtain the spectrum.

[0059] In this embodiment, a minimum variance distortion-free response (MVDR) beamformer is used in step S21. The MVDR beamformer operates in a narrowband. Step S12 involves processing the time-domain snapshot data matrix. Taking the Fourier transform and extracting the positive frequency components yields... A narrow band. The following calculation process requires consideration of all... A narrow-band repeat. MVDR beamformer in UAV candidate directions. and frequency The weight vector at that point is

[0060] Noise covariance matrix It consists of two parts: one part comes from candidate directions other than drones. The covariance matrix of the plane waves in all other candidate directions of the UAV is given by one part, and the covariance matrix of Gaussian white noise distributed throughout space is given by another part. The two parts are summed. The amplitude of the plane waves in the other candidate directions of the UAV can be taken as the arithmetic square root of the corresponding local maxima of the spatial power spectrum.

[0061] S22. For each candidate direction of the UAV, use the anomaly detector trained in the offline part to determine whether the extracted acoustic signal spectrum is caused by the UAV. S23. For each candidate direction of the UAV, select an image patch on the candidate region image of the first camera, and use the target detector trained in the offline part to determine whether the extracted image patch contains the UAV.

[0062] In this embodiment, the resolution of the image block is set to 96×96. The candidate direction for the UAV is set as follows: For a selected image patch on the candidate region image of the first camera, let the coordinates of its center on the candidate region image be... The offline camera calibration yielded the camera intrinsic parameter matrix. It contains , , and These parameters can be used to express and Relationship:

[0063] S24. All candidate drone directions that were determined to be "true" in steps S22 and S23 are added to the drone direction set. The drone's orientation is represented by coordinates. It indicates. Among them, It is the angle between the line connecting the drone and the center of the acoustic array and the ground. It is the projection of the line connecting the UAV and the center of the acoustic array onto the ground plane and the array plane. The angle between the positive axis and the positive axis.

[0064] Specifically, such as Figure 8 As shown, step S3 includes the following sub-steps: S31. Acquire images of the detection area using a binocular camera, and use the epipolar correction parameters obtained from the offline partial camera calibration to perform epipolar correction on the two images, so that the object points in the real world are in the same row on the two images. S32. Based on the two images that have been epipolarized, use the deep learning-based binocular disparity predictor trained in the offline part to predict the disparity value of each point in the reference camera image.

[0065] In engineering applications, performing neural network inference on original-size images with a resolution of 1920×1080 consumes significant computational resources and substantially reduces system response speed. Without significantly sacrificing ranging accuracy, the epipolar-corrected image can be downsampled by a factor of 2 to achieve a resolution of 960×540, and the predicted disparity can be multiplied by 2.

[0066] S33. Using the baseline distance obtained from the offline partial camera calibration. and focal length disparity value Convert to depth value :

[0067] Specifically, step S4 will determine the drone's location. A unique location corresponding to a depth image The method is

[0068] in, , , and All parameters are derived from the camera intrinsic parameter matrix obtained from the offline partial camera calibration. .

[0069] Specifically, step S5 combines the UAV location obtained in step S2. and the drone distance obtained in step S4 Obtain the three-dimensional coordinates of the UAV for .

[0070] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting and locating unmanned aerial vehicles (UAVs) through audiovisual fusion, characterized in that, include: S1, receive acoustic signals from the detection area through an acoustic array, construct a spatial power spectrum based on the acoustic signals, and search for the location in the detection area where the acoustic signals are strongest locally in the spatial power spectrum, as a candidate location set for the UAV; S2, image blocks of each candidate drone location are acquired by a binocular camera, and drone location is determined to be occupied by drones based on drone sound signal anomaly detector and drone image target detector. When the output of drone sound signal anomaly detector and drone image target detector are both "yes", it is determined that drones are occupied by drones in the candidate drone location. All drone candidate locations with drones are formed into a drone location set. S3, predict the disparity of each point in the reference camera image in the binocular camera using a binocular disparity predictor, and convert the disparity value into a depth value using the baseline distance and focal length obtained by the binocular camera calibration in advance, to obtain a depth image from the perspective of the reference camera. S4. Based on the center alignment relationship between the acoustic array and the reference camera, the azimuth coordinates of the UAV are mapped to the corresponding position in the depth image. At this time, the depth value at that position is used as the vertical distance of the UAV. S5 establishes three-dimensional coordinates based on the drone's orientation and vertical distance, thereby obtaining the drone's three-dimensional coordinates and enabling drone identification and spatial positioning.

2. The audiovisual fusion drone detection and positioning method according to claim 1, characterized in that, Step S1 specifically includes the following steps: S11, receiving acoustic signals from the detection area via an acoustic array, wherein the acoustic array includes Each microphone, all microphones with Sampling frequency synchronous sampling Next, obtain the temporal snapshot data matrix. ,matrix Include lines and List real numbers, that is, numbers with the shape of and ;in, Represents the set of real numbers; S12, for the matrix Perform a Fast Fourier Transform on each row and extract the positive frequency components to obtain the frequency domain snapshot data matrix. ,matrix The elements belong to the set of complex numbers. ,Right now ;in, ;matrix The OK( ), No. List( The element of ) represents the first element. In the signal received by each microphone, the frequency The amplitude and phase; where , Indicates rounding down; S13, Select center frequency That is, selecting a matrix The The column yields the frequency domain snapshot data vector. ,and ; S14, Select Each acoustic scanning point has a line connecting it to the center of the acoustic array, representing the direction of the incoming acoustic signal. The main lobe of the beamformer is adjusted sequentially to scan all the points. The direction of the incoming wave is used to obtain the beamformer. Each response outputs a value, then the power is calculated; all Each power constitutes a spatial power spectrum; S15, Search for local maxima on the spatial power spectrum. The values ​​of the local maxima should be higher than a preset threshold. The incoming wave directions of all local maxima corresponding to the scanning points constitute the candidate azimuth set of the UAV.

3. The audiovisual fusion drone detection and positioning method according to claim 2, characterized in that, The beamformer in step S14 is in the... The response output in each direction is in, It is the beamformer at the frequency , No. Weight vectors in each direction.

4. The audiovisual fusion drone detection and positioning method according to claim 1, characterized in that, Step S2 specifically includes the following steps: S21, for each candidate azimuth of the UAV, the main lobe of the beamformer is adjusted to that direction, and the beamformer is required to suppress the signals of all other candidate azimuths of the UAV, so as to sequentially receive the acoustic signals of each candidate azimuth of the UAV and obtain the spectrum. S22, For each candidate drone location, a deep learning-based drone acoustic signal anomaly detector is used to determine whether the extracted acoustic signal spectrum is caused by a drone. S23, for each candidate drone location, select an image patch on the candidate region image of the reference camera, and use a deep learning-based drone image target detector to determine whether the extracted image patch contains a drone; S24, all candidate drone locations that were determined to be "yes" in both steps S22 and S23 are added to the drone location set. The drone location is represented by coordinates. Indicates; among which, It is the angle between the line connecting the drone and the center of the acoustic array and the ground. It is the projection of the line connecting the drone and the center of the acoustic array onto the ground, and the ground-preset... The angle between directions.

5. The audiovisual fusion unmanned aerial vehicle (UAV) detection and positioning method according to claim 4, characterized in that, Step S21 selects a minimum variance distortion-free response beamformer, i.e., an MVDR beamformer, which is used in the candidate azimuth of the UAV. and frequency The weight vector at that point is in, It is the noise covariance matrix. The acoustic array is related to the candidate azimuth of the drone. of, in frequency manifold vectors on.

6. The audiovisual fusion UAV detection and positioning method according to claim 4, characterized in that, Step S23: Candidate UAV location The coordinates of the center of the selected image patch on the candidate region image of the reference camera. The relationship is in, , , and All data are derived from the camera intrinsic parameter matrix obtained from camera calibration. .

7. The audiovisual fusion drone detection and positioning method according to claim 1, characterized in that, Step S3 specifically includes the following steps: S31, images of the detection area are acquired by a binocular camera, and the epipolar correction parameters obtained from camera calibration are used to perform epipolar correction on the two images, so that the object points in the real world are in the same row on the two images. S32, based on two epipolar-corrected images, uses a deep learning-based binocular disparity predictor to predict the disparity values ​​of each point in the reference camera image; S33 uses the baseline distance and focal length obtained from camera calibration to convert the parallax value into a depth value: in, and These are the x-coordinates of the image points of the real-world object points as perceived by the reference camera and the second camera, respectively. This is parallax; It is the depth value of the object point from the perspective of the reference camera. It is the straight-line distance between the optical centers of the reference camera and the second camera, i.e., the baseline distance; It refers to the focal length of the two cameras.

8. The audiovisual fusion unmanned aerial vehicle (UAV) detection and positioning method according to claim 1, characterized in that, Step S4 according to the formula Positioning of the drone A unique location corresponding to a depth image ,in, , , and All data are derived from the camera intrinsic parameter matrix obtained from camera calibration. .

9. The audiovisual fusion unmanned aerial vehicle (UAV) detection and positioning method according to claim 1, characterized in that, 3D coordinates of the drone for ,in Location of the drone. The vertical distance for the drone.

Images