A three-dimensional stereovision and sound composite detection system, device and computer storage medium

Abstract

This invention discloses a three-dimensional stereoscopic audio-visual composite detection system, device, and computer storage medium. The system includes a three-dimensional stereoscopic microphone array, a rotatable camera unit, a multi-channel signal acquisition and processing unit, and a multimodal data fusion processing module. The three-dimensional stereoscopic microphone array consists of six arms, with four horizontal arms arranged radially in a cross shape in the horizontal plane, and two vertical arms arranged side by side in the central vertical direction; each arm contains 48 microphones, forming a three-dimensional sampling grid, with an overall footprint of 213cm × 213cm and a height of 111cm. The rotatable camera unit is mounted at the top of the vertical arm and has a 360° horizontal and pitch-adjustable rotation capability. The multimodal data fusion processing module detects and identifies targets based on intelligent algorithms, and uses the Time Difference of Arrival (TDOA) algorithm to calculate the three-dimensional coordinates of the sound source, guiding the camera unit to perform visual verification, and realizing the fusion output of the target's three-dimensional coordinates and image information. This invention can achieve audio-visual composite detection of low-altitude targets, with high positioning accuracy, strong anti-interference ability, and good real-time performance, and is suitable for fields such as UAV detection, key area defense, and border monitoring.

Description

Technical Field

[0001] This invention belongs to the field of acoustic and visual composite detection technology, specifically relating to a three-dimensional stereoscopic acoustic-visual composite detection system and its detection method for low-altitude target detection. Background Technology

[0002] With the popularization of drone technology, the safety hazards posed by low-altitude aircraft are becoming increasingly prominent. Traditional radar detection methods have inherent shortcomings in detecting low-altitude, slow-moving, and small targets: low-altitude aircraft fly at low altitudes, have small radar cross-sections, and are subject to ground clutter interference, making stable radar detection difficult. Acoustic detection, as a passive detection method, has advantages such as low cost, good concealment, and immunity to electromagnetic interference, and can effectively compensate for the blind spots of radar detection.

[0003] However, acoustic detection alone still has limitations in target recognition and environmental adaptability; while visual detection is greatly affected by factors such as light and occlusion, making it difficult to achieve all-weather detection independently. Therefore, there is an urgent need for a three-dimensional spatial target detection system that can integrate the advantages of acoustics and vision to achieve high precision, anti-interference, and all-weather operation. Summary of the Invention

[0004] Purpose of the invention This invention aims to solve the problems of insufficient dimensionality, weak anti-interference ability, and poor environmental adaptability of existing single detection methods, and provides a three-dimensional stereoscopic audio-visual composite detection system, device and computer storage medium to achieve high-precision three-dimensional detection and identification of low-altitude targets such as UAVs. Technical solution

[0005] This invention provides a three-dimensional stereoscopic audio-visual composite detection system, comprising:

[0006] (1) Three-dimensional stereo microphone array A three-dimensional coordinate support structure is adopted, with microphones arranged along the X, Y, and Z spatial axes to form a three-dimensional sampling grid. Specifically, the array consists of six arms: four horizontal arms are evenly distributed radially in a cross shape in the horizontal plane, and the other two vertical arms are arranged side by side in the central vertical direction. For easy distinction, in this embodiment, the four horizontal arms are designated as Arm 1, Arm 2, Arm 3, and Arm 4, and the two vertical arms are designated as Arm 5 and Arm 6. Each horizontal arm consists of two cover plates, with four microphones mounted on each cover plate, for a total of eight microphones per horizontal arm and 32 microphone channels across the four horizontal arms; each vertical arm has eight microphones mounted on it, for a total of 16 microphone channels across the two vertical arms; the entire array has a total of 48 microphone channels. The horizontal arms are 101cm long and 2.2cm wide; the overall array dimensions are 213cm × 213cm, and the height is 111cm. The microphones are evenly distributed within the support frame, with an element spacing of 10cm. The base support includes a base disc (10cm in diameter) and a height adjustment block (7.5cm×2cm×5cm) for adjusting the overall height of the array.

[0007] (2) Rotatable camera unit Mounted at the top of two vertically aligned arms arranged side-by-side in the center, this integrated pan-tilt camera unit features a wide-angle lens and autofocus, offering 360° continuous horizontal rotation and -20° to +90° tilt adjustment. Guided by acoustic detection, this unit enables rapid visual verification of target areas, facilitating target type confirmation and tracking.

[0008] (3) Multi-channel signal acquisition and processing unit It is electrically connected to each microphone in a three-dimensional stereo microphone array to synchronously acquire multi-channel sound pressure signals and perform analog-to-digital conversion, filtering, amplification, and preprocessing.

[0009] (4) Multimodal data fusion processing module Receive preprocessed multi-channel sound pressure signals and image data acquired by the rotatable camera unit, and perform the following functions: Sound source localization: Calculating the three-dimensional spatial coordinates of the target based on multi-channel sound pressure signals; Visual guidance: Based on the acoustic positioning results, control the rotatable camera unit to rotate to the corresponding direction; Image recognition: Detecting and recognizing targets in acquired images; Data fusion: The acoustic positioning results are fused with the image recognition results to output the three-dimensional spatial coordinates and image information of the target.

[0010] Compared with the prior art, the present invention has the following beneficial effects:

[0011] (1) Audio-visual fusion and information complementarity: Wide-area passive detection is achieved through acoustic array, and target accurate confirmation is achieved through rotatable camera unit, which effectively makes up for the limitations of a single sensor and improves the system's detection capability in complex environments.

[0012] (2) Three-dimensional spatial positioning: The three-dimensional stereo microphone array can simultaneously acquire the target's azimuth, elevation, and distance information, achieving true three-dimensional spatial positioning and effectively compensating for the insufficient resolution of traditional planar arrays in the height direction. The measured angle positioning accuracy can reach the sub-degree level, meeting the practical requirements of low-altitude UAV detection.

[0013] (3) Guided visual verification: The acoustic detection results guide the camera unit to quickly point to the target area, which improves the response speed and target acquisition efficiency of the vision system, reduces the dependence on omnidirectional high-definition video, and reduces system costs.

[0014] (4) Strong anti-interference capability: The adaptive anti-interference unit can effectively suppress environmental noise and improve the positioning robustness under low signal-to-noise ratio conditions.

[0015] (5) Reasonable structural design: The microphone array with equal spacing and uniform distribution has a simple structure and is easy to manufacture; the camera unit is installed at the top of the vertical arm to form an integrated detection platform, which is convenient for deployment and maintenance. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the overall structure of the three-dimensional stereoscopic audio-visual composite detection system of the present invention;

[0017] Figure 2 This is a schematic diagram of the structure of the three-dimensional microphone array of the present invention;

[0018] Figure 3 This is a flowchart illustrating the workflow of the multimodal data fusion processing module of the present invention.

[0019] Figure 4 This is a schematic diagram illustrating the application effect of the present invention in a drone detection scenario. Detailed Implementation

[0020] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0021] Example 1: Construction of a 3D Stereo Microphone Array and a Rotatable Camera Unit like Figure 2As shown, the three-dimensional microphone array in this embodiment includes a bottom bracket and a three-dimensional coordinate support frame mounted on top of the bracket. The support frame consists of six arms, of which arms 1, 2, 3, and 4 are horizontal arms, evenly distributed radially in a cross shape in the horizontal plane; arms 5 and 6 are vertical arms, arranged side by side in the central vertical direction. Each horizontal arm consists of two cover plates, with four microphones mounted on each cover plate, forming a total of 32 microphone channels; each vertical arm has eight microphones mounted, forming a total of 16 microphone channels. The entire array has a total of 48 microphone channels. The horizontal arms are 101cm long and 2.2cm wide; the overall dimensions of the array are 213cm × 213cm, and the height is 111cm.

[0022] The microphones are evenly distributed at equal intervals inside the support frame, with an element spacing of 10cm. The bottom support includes a base disc (10cm in diameter) and a height-adjusting block (7.5cm×2cm×5cm) for adjusting the overall height of the array.

[0023] The rotatable camera unit is an integrated pan-tilt camera, mounted on the top of two vertically aligned arms (arms 5 and 6) arranged side-by-side in the center. It has the capability of continuous 360° horizontal rotation and pitch adjustment from -20° to +90°. In this embodiment, this camera unit is labeled "camera unit" in the figure. The entire system is painted black to reduce ambient light reflection interference.

[0024] Example 2: Multi-channel signal acquisition and processing In this embodiment, the multi-channel signal acquisition and processing unit synchronously acquires 48 channels of microphone signals at a sampling frequency of 96kHz, with a quantization accuracy of 16 bits. The acquired signals are filtered, amplified, and gain calibrated to eliminate amplitude and phase inconsistencies between channels. The sample-and-hold and analog-to-digital conversion of each microphone channel are triggered by a unified clock, ensuring that the time synchronization accuracy between channels is better than 1 microsecond.

[0025] Example 3: Multimodal Data Fusion Processing In this embodiment, the multimodal data fusion processing module performs the following steps:

[0026] Step 1: Sound source localization. Based on multi-channel sound pressure signals, the acoustic three-dimensional spatial coordinates of the target are calculated using acoustic localization methods.

[0027] Step 2: Visual guidance. Based on the acoustic positioning results, calculate the required rotation angle of the rotatable camera unit and drive the gimbal to point the camera in the target direction.

[0028] Step 3: Image Acquisition and Recognition. The camera unit acquires images of the target area, identifies the target type in the image through a target detection algorithm, and locks onto the target.

[0029] Step 4: Data Fusion Output. The 3D coordinates obtained from acoustic localization are correlated with the target category information from image recognition, and the fused detection results are output.

[0030] Example 4: Adaptive Anti-interference Processing In this embodiment, the multimodal data fusion processing module also includes an adaptive anti-interference unit. This unit employs an adaptive filtering algorithm based on minimum mean square error (LMS) to estimate the autocorrelation matrix of environmental noise in real time, calculate the optimal filtering coefficients, and perform noise reduction processing on the acquired signals. This module can effectively suppress interference such as wind noise and urban background noise, and maintain stable positioning performance even under low signal-to-noise ratio conditions.

[0031] Example 5: Application in UAV Detection like Figure 4 As shown, this invention can be deployed on the ground or a communication tower for the detection and tracking of low-altitude unmanned aerial vehicle (UAV) targets. The system uses an acoustic array to achieve passive long-range detection and 3D positioning of the UAV. After obtaining the target's direction, it guides a rotatable camera unit to rotate to the target area for visual confirmation and tracking. Real-world testing shows that the system achieves sub-degree-level angle positioning accuracy in typical environments, meets real-time response requirements, and can effectively handle multi-target scenarios.

Claims

1. A three-dimensional stereoscopic audio-visual composite detection system, characterized in that, include: A three-dimensional stereo microphone array, comprising a bottom bracket and a three-dimensional coordinate shape support frame mounted on the bottom bracket, the support frame consisting of six arms, of which four horizontal arms are evenly distributed radially in a cross shape in a horizontal plane, and the other two vertical arms are arranged side by side in the central vertical direction; each arm contains multiple microphones, forming a three-dimensional sampling grid. A rotatable camera unit is installed at the top of two vertical arms arranged side by side in the central vertical direction. The rotatable camera unit has a 360° horizontal rotation capability and an adjustable pitch capability, and is used to achieve visual verification of the target area under acoustic detection guidance. The multi-channel signal acquisition and processing unit is electrically connected to each microphone in the three-dimensional microphone array and is used to synchronously acquire multi-channel sound pressure signals and perform preprocessing. The multimodal data fusion processing module receives pre-processed multi-channel sound pressure signals and image data acquired by the rotatable camera unit, performs target recognition and spatial position calculation, and outputs the target's three-dimensional spatial coordinates and image information.

2. The three-dimensional stereoscopic audio-visual composite detection system according to claim 1, characterized in that, The four horizontal arms are evenly distributed in a cross-shaped radial pattern within the horizontal plane. Each horizontal arm consists of two cover plates, and four microphones are installed on each cover plate, forming a total of 32 microphone channels. The two vertical arms arranged side by side in the central vertical direction each have 8 microphones installed on them, forming a total of 16 microphone channels; The entire array has a total of 48 microphone channels.

3. The three-dimensional stereoscopic audio-visual composite detection system according to claim 2, characterized in that, Each horizontal arm in the horizontal plane is 101cm long and 2.2cm wide; the overall size of the array is 213cm × 213cm and the height is 111cm.

4. The three-dimensional stereoscopic audio-visual composite detection system according to claim 1, characterized in that, The bottom support includes a base disc and a height-adjusting block. The base disc has a diameter of 10cm, and the height-adjusting block has dimensions of 7.5cm × 2cm × 5cm (length × width × height), and is used to adjust the overall height of the array.

5. The three-dimensional stereoscopic audio-visual composite detection system according to claim 1, characterized in that, The multiple microphones are evenly distributed at equal intervals inside the support frame, with an array element spacing of 10cm.

6. The three-dimensional stereoscopic audio-visual composite detection system according to claim 1, characterized in that, The rotatable camera unit is an integrated pan-tilt camera with a built-in wide-angle lens and autofocus function. Its rotation is controlled by the multimodal data fusion processing module based on the sound source localization results.

7. The three-dimensional stereoscopic audio-visual composite detection system according to claim 1, characterized in that, The multimodal data fusion processing module includes a sound source localization calculation unit and an image analysis unit. The sound source localization calculation unit obtains the three-dimensional spatial coordinates of the target based on multi-channel sound pressure signals. The image analysis unit guides the rotatable camera unit to rotate to the corresponding direction according to the three-dimensional spatial coordinates and performs target confirmation and tracking based on the acquired images.

8. The three-dimensional stereoscopic audio-visual composite detection system according to claim 1, characterized in that, The multimodal data fusion processing module also includes an adaptive anti-interference unit, which contains an adaptive filter for noise reduction of the acquired signal.

9. A three-dimensional stereoscopic audio-visual composite detection method based on the system described in any one of claims 1-8, characterized in that, Includes the following steps: Step S1: Simultaneously acquire multi-channel sound pressure signals through the three-dimensional stereo microphone array, and simultaneously acquire initial environmental images through the rotatable camera unit; Step S2: Preprocess the acquired sound pressure signal, including filtering and amplification, analog-to-digital conversion, and channel consistency calibration; Step S3: Based on the preprocessed sound pressure signal, calculate the acoustic three-dimensional spatial coordinates of the target using an acoustic localization method; Step S4: Based on the acoustic three-dimensional spatial coordinates, control the rotatable camera unit to rotate to the target direction and acquire a high-definition image of the target area; Step S5: Fuse the acoustic positioning results with the image data to output the target's three-dimensional spatial coordinates and image information.

Images