A sound ball linkage abnormal sound positioning method

Abstract

The application provides a sound ball linkage abnormal sound positioning method, first, air sonar is arranged beside a monitoring ball camera; then, the air sonar continuously collects sound signals and detects abnormal sound, and the sound source position of the abnormal sound is calculated; finally, the ball camera is guided to turn to the direction of the abnormal sound. Through improving the linkage technology of the sonar array and the ball camera, the response time of the ball camera to the abnormal sound source is greatly reduced, only the relative position of the sonar array and the camera needs to be calibrated, the workload is reduced, and the linkage precision is improved.

Description

Technical Field

[0001] This invention relates to the field of video surveillance, and in particular to a sound localization method. Background Technology

[0002] With the development of science and computer technology, and the rapid progress of artificial intelligence in recent years, the means by which humans and cities obtain information from the outside world have become increasingly diversified, and local governments have proposed development plans for smart cities. However, urban and industrial monitoring still relies on relatively singular methods for acquiring information flow.

[0003] How to compensate for lost audio information in video surveillance, and to improve the city's "hearing" by integrating environmental acoustics and optical information for joint decision-making, is of great significance for improving the city's sensory system and promoting the construction of intelligent cities.

[0004] Current technologies generally monitor the environment through cameras. However, cameras are prone to missing detections. Although there are technologies that use sonar arrays and PTZ cameras for monitoring, most existing camera and sonar array linkage technologies use a preset camera point scheme.

[0005] Microphone arrays can detect horizontal planes within a range of 20m to 50m. According to traditional linkage schemes, multiple sets of data need to be collected within the 20m to 50m range and a large number of preset points need to be set, which seriously wastes manpower and resources during implementation. On the other hand, the functional relationship obtained by fitting the data has errors, including measurement errors and system errors, which leads to inaccurate positioning of the camera based on audio linkage and poor performance in practical applications.

[0006] The patent "A Method for Monitoring Abnormal Sound Sources in Linkage Between a Microphone Array and a PTZ Camera" first detects abnormal data in the microphone array. Then, through a sound source localization module, it obtains the horizontal azimuth, pitch, and distance of the sound source relative to the microphone array plate. Based on these parameters, it calculates the sound source's position relative to the microphone array. By selecting three reference points in the video frame and measuring their positions relative to the camera, a world coordinate system is established. Using camera imaging principles, the world coordinates of the three reference points are transformed into pixel coordinates and displayed on the screen. When the displayed position coincides with the actual position of the object, camera calibration is complete, obtaining the camera's preset attitude. This allows for the determination of the required horizontal rotation and pitch angles of the PTZ camera, achieving PTZ camera linkage. However, this patented solution, by establishing a unified spatial coordinate system, requires more computational resources. Furthermore, calculation errors occur during multiple coordinate transformations, leading to inaccurate calculations of the PTZ camera's rotation angles and insufficient real-time performance, resulting in poor practical application performance. Summary of the Invention

[0007] To overcome the shortcomings of existing technologies, this invention provides a method for locating abnormal sounds through acoustic ball linkage, which can effectively improve the measurement accuracy and real-time performance.

[0008] The technical solution adopted by this invention to solve its technical problem includes the following steps:

[0009] Step 1: Place the air sonar next to the monitoring PTZ camera;

[0010] Step 2: Continuously collect sound signals and detect abnormal sounds using air sonar;

[0011] Step 3: Calculate the location of the source of the abnormal sound;

[0012] Step 4: Guide the PTZ camera to turn in the direction of the abnormal sound source based on the location of the sound source.

[0013] Step 1 involves placing the air sonar within a 50cm range above or below the monitoring PTZ camera.

[0014] Step 2 includes the following steps:

[0015] (1) Real-time calculation of environmental characteristic data;

[0016] (2) Calculate the channel number with the highest sound decibel value in the current frame and the related environmental feature data;

[0017] (3) Determine whether the highest value of the sound decibel in the current frame is greater than the preset value of the set decibel threshold. If it is greater, proceed to the next step; otherwise, return to step (1).

[0018] (4) Calculate the temporal energy ratio of the sound in the current frame to that in the previous frame;

[0019] (5) Determine whether the time-domain energy ratio is greater than the preset threshold. If it is, proceed to the next step; otherwise, return to step (1).

[0020] (6) Calculate the frequency domain energy ratio of the sound in the current frame;

[0021] (7) Determine whether the frequency domain energy ratio is greater than the preset threshold. If it is, trigger an alarm; otherwise, return to step (1).

[0022] The frequency domain energy ratio is used to determine the ratio of the energy of the detection frequency band to the total frequency band. The adjustment range of the lower limit of the detection frequency band is 0 to 10 kHz, and the numerical input must be an integer and less than the upper limit. The adjustment range of the upper limit of the detection frequency band is 0 to 10 kHz, and the numerical input must be an integer and greater than the lower limit. The difference between the lower limit and the upper limit of the detection frequency band is greater than 100 Hz.

[0023] Step 3 utilizes array spatial information and conventional beamforming technology to locate the sound source, calculates the sound source direction information, and sends it to the monitoring PTZ camera.

[0024] The sound source localization steps are as follows: sound source localization is performed by beamforming; N sound sensors form a receiving array; for each received signal x... i (t) Select the set weighting vector w i (θ) is weighted and summed to obtain the output y(t,θ) of the array; assuming the target signal is a narrowband signal with a center frequency of f, then the input for beamforming... The vector of sampled data and complex weighting coefficients for all array elements is represented as x(t) = [x1(t) x2(t)...x N (t)] T and w(θ)=[w1(θ) w(θ)…w N (θ)] T The beamforming output, expressed in the form of a vector dot product, is y(t, θ) = w H (θ)x(t)=x H (t)w(θ), where the superscript * denotes the complex conjugate operator, the superscript T denotes the transpose of a vector or matrix, and the superscript H denotes the complex conjugate transpose of a vector or matrix; the power spectrum of the beamforming output is expressed as P(θ)=E[|y(t,θ)| 2 ] = w H (θ)Rw(θ); Theoretically, the deflection angle of the sound source relative to the normal of the center of the planar array is (0°, 0°). Using a 3000Hz single-frequency signal as the sound source, a simulation of conventional planar beamforming of a 16-element array is performed under the condition of a receiving signal-to-noise ratio of 0dB.

[0025] In step 4, after the monitoring PTZ camera is started, zero-point calibration is performed to match the 0° position of the monitoring PTZ camera with the 0° position of the air sonar; then, direction calibration is performed to ensure that the rotation direction of the verification PTZ camera is consistent with the direction marked by the sonar; the coordinate information is converted into the rotation angle of the monitoring PTZ camera to guide the monitoring PTZ camera to turn to the direction of abnormal sound; the monitoring PTZ camera continuously captures the direction of abnormal sound and stores it together with the alarm recording as an alarm log.

[0026] Step 4 involves continuously capturing images at 1-second intervals and saving the audio data five seconds before and after the alarm as alarm recordings, which are then stored together as an alarm log.

[0027] The beneficial effects of this invention are: by improving the linkage technology between the sonar array and the PTZ camera, the response time of the PTZ camera to abnormal sound sources is greatly reduced. The previously cumbersome conversion of multiple coordinates is now simplified to simply calibrating the relative positions of the sonar array and the camera, thus reducing workload and improving linkage accuracy. During operation, the security command center sends instructions to each node, controlling the nodes to modify detection thresholds to complete sonar calibration. This calibration process can be completed uniformly before sonar deployment or adjusted appropriately based on the site environment after deployment. Prioritizing the configuration of decibel thresholds, time-domain energy ratio thresholds, frequency band energy ratio thresholds, and upper and lower limits of the detection frequency band enables rapid use of the sonar. Attached Figure Description

[0028] Figure 1 This is a flowchart of the method of the present invention;

[0029] Figure 2 This is a flowchart of the abnormal sound alarm process of the present invention;

[0030] Figure 3 This is a flowchart of the abnormal sound localization process of the present invention;

[0031] Figure 4 This is a flowchart illustrating the process of guiding the PTZ camera to capture images according to the present invention. Detailed Implementation

[0032] The present invention will be further described below with reference to the accompanying drawings and embodiments. The present invention includes, but is not limited to, the following embodiments.

[0033] like Figure 1 As shown, the present invention provides a method for locating and tracking abnormal sounds in a spherical linkage system, comprising the following steps:

[0034] Step S300: First, place the air sonar within 50cm above or below the existing monitoring PTZ camera;

[0035] Step S310: Continuously collect sound signals and detect abnormal sounds using air sonar;

[0036] Step S320: The sonar front-end processor calculates the location of the source of the abnormal sound;

[0037] Step S330: Guide the PTZ camera to turn in the direction of the abnormal sound source based on the location of the sound source.

[0038] In step S300, an air sonar array is first installed near the existing video surveillance equipment. After installation, it forms a smart monitoring node and is centralized to the security monitoring center through a switch. The sound sensor uses air sonar and is set above or below the monitoring equipment to collect the sound signal emitted by the sound source.

[0039] according to Figure 2The diagram shows a flowchart of an abnormal sound alarm according to an embodiment of the present invention. This diagram illustrates step S310 of the acoustic ball linkage abnormal sound location and tracking method, which specifically includes the following steps:

[0040] Step S310: After the sonar array starts working, proceed to step S311 to calculate environmental feature data per second in real time.

[0041] In step S312, the device will calculate the channel number of the highest sound decibel value and related environmental characteristic data in the current frame (0.1s) every second via the HTTP protocol;

[0042] Step S313: Determine whether the highest decibel value of the sound in the current frame is greater than the preset value of the decibel threshold (70dB). If it is greater, proceed to step S314; otherwise, continue to execute S311.

[0043] Step S314: The device calculates the temporal energy ratio of the sound in the current frame to that in the previous frame via the HTTP protocol to determine if there is a sudden loud noise in the environment.

[0044] Step S315: Determine whether the time-domain energy ratio is greater than a preset threshold. If it is, proceed to step S316; otherwise, continue with step S311.

[0045] Step S316: When the highest decibel value and the time-domain energy ratio are both greater than the preset threshold, calculate the current frequency-domain energy ratio to determine the ratio of the energy of the detection frequency band to the total frequency band.

[0046] Among them, the lower limit of the detection frequency band is used to adjust the lower limit of the frequency band for environmental detection. The adjustable range is 0 to 10KHz. The numerical input must be an integer and less than the upper limit. The difference must be greater than 100Hz.

[0047] Upper limit of detection frequency band: Used to adjust the upper limit of the frequency band for environmental detection. The adjustable range is 0 to 10 kHz. The numerical input must be an integer and greater than the lower limit. The difference must be greater than 100 Hz.

[0048] In step S317, if the frequency domain energy ratio is greater than the preset value, proceed to step S318 to trigger an alarm; otherwise, continue executing S311.

[0049] like Figure 3 As shown, in step S320, when the system detects sensitive noise, step S321 is executed. The system immediately uses array spatial information and conventional beamforming technology to locate the sound source and calculate the sound source direction information. Then, step S322 is executed, and the sound source information is sent to the PTZ camera via HTTP protocol. The sonar front-end processor calculates the location of the abnormal sound source using conventional beamforming, and the specific steps are as follows;

[0050] Once an abnormal sound signal is detected, beamforming is used to locate the sound source. N sound sensors form a receiving array, and each received signal x... i (t) Only the set weighting vector w needs to be selected. i (θ) Then, by performing a weighted summation, we can obtain the output y(t,θ) of the array. Assuming the target signal is a narrowband signal with a center frequency of f, the input for beamforming can be expressed as:

[0051]

[0052] The vector representation of the sampled data of all array elements and the complex weighting coefficients is as follows:

[0053] x(t)=[x1(t) x2(t)…x N (t)] T

[0054] w(θ) = [w1(θ) w(θ)…w N (θ)] T

[0055] The beamforming output can then be expressed in the form of a vector inner product as follows:

[0056] y(t, θ) = w H (θ)x(t)=x H (t)w(θ)

[0057] In the formula, the superscript * denotes the complex conjugate operator, the superscript T denotes the transpose of a vector or matrix, and the superscript H denotes the complex conjugate transpose of a vector or matrix.

[0058] The power spectrum at the output of the beamforming can be expressed as:

[0059] P(θ) = E[|y(t, θ)| 2 ] = w H (θ)Rw(θ)

[0060] Theoretically, the deflection angle of the sound source relative to the center normal of the planar array should be (0°, 0°). Using a 3000Hz single-frequency signal as the sound source, a simulation of conventional planar beamforming of a 16-element array is performed under the condition of a receiving signal-to-noise ratio of 0dB.

[0061] like Figure 4As shown, in step S330, after the PTZ camera starts up, step S331 is executed first to perform zero-point calibration: this function is used to adapt the PTZ camera's 0° position to the sonar's 0° position, ensuring that the direction of the sound source identified by the sonar is consistent with the direction the PTZ camera points after rotation. Then, step S332 is executed to perform direction calibration: this function is used to verify whether the PTZ camera's rotation direction is consistent with the sonar's marked direction. After triggering, the PTZ camera rotates counterclockwise. If they are consistent, the sonar outputs 100 degrees, and the PTZ camera rotates to 100 degrees; if they are inconsistent, the sonar outputs 100 degrees, and the PTZ camera rotates to 260 degrees. After the PTZ camera information is set correctly, the PTZ camera receives the sound source position calculated in step S322 and proceeds to step S333, converting the coordinate information into the PTZ camera's rotation angle. Through local network control, the PTZ camera is guided to turn towards the direction of the abnormal sound. Immediately following step S334, it is determined whether the PTZ camera continuously captures the abnormal sound. If so, it performs continuous captures at 1-second intervals. Finally, in step S335, the system saves the sound data for five seconds before and after the alarm as a ten-second alarm recording. By default, the system will store all alarm logs.

[0062] This invention significantly reduces the response time of the PTZ camera to abnormal sound sources by improving the linkage technology between the sonar array and the PTZ camera. The previously cumbersome conversion of multiple coordinates is now reduced to simply calibrating the relative positions of the sonar array and the camera, thus reducing workload and improving linkage accuracy. The security command center can quickly deploy the sonar by sending instructions to each node to prioritize and configure detection thresholds. The location of the abnormal sound source processed by the sonar front-end is converted into the rotation angle of the PTZ camera. Local network control guides the PTZ camera to turn in the direction of the abnormal sound for further observation and processing. This allows for sound source localization of abnormal sounds in milliseconds. After an alarm event occurs, audio file evidence of several seconds before and after the event is accurately stored, eliminating the need for massive data review and retrieval.

Claims

1. A method for locating abnormal sounds in a sphere-linked acoustic system, characterized in that, Includes the following steps: Step 1: Place the air sonar next to the monitoring PTZ camera; Step 2 involves continuously acquiring sound signals and detecting abnormal sounds using an airborne sonar; Step 2 includes the following steps: (1) Real-time calculation of environmental characteristic data; (2) Calculate the channel number with the highest sound decibel value in the current frame and the related environmental feature data; (3) Determine whether the highest value of the sound decibel in the current frame is greater than the preset value of the set decibel threshold. If it is greater, proceed to the next step; otherwise, return to step (1). (4) Calculate the temporal energy ratio of the sound in the current frame to that in the previous frame; (5) Determine whether the time-domain energy ratio is greater than the preset threshold. If it is, proceed to the next step; otherwise, return to step (1). (6) Calculate the frequency domain energy ratio of the sound in the current frame; (7) Determine whether the frequency domain energy ratio is greater than the preset threshold. If it is, trigger an alarm; otherwise, return to step (1). Step 3: Calculate the location of the source of the abnormal sound; Step 4: Guide the PTZ camera to turn in the direction of the abnormal sound source based on the location of the sound source.

2. The method for locating abnormal sounds in a sphere-linked acoustic system according to claim 1, characterized in that, Step 1 involves placing the air sonar within a 50cm range above or below the monitoring PTZ camera.

3. The method for locating abnormal sounds in a sphere-linked acoustic system according to claim 2, characterized in that, The frequency domain energy ratio is used to determine the ratio of the energy of the detection frequency band to the total frequency band. The adjustment range of the lower limit of the detection frequency band is 0 to 10 kHz, and the numerical input must be an integer and less than the upper limit. The adjustment range of the upper limit of the detection frequency band is 0 to 10 kHz, and the numerical input must be an integer and greater than the lower limit. The difference between the lower limit and the upper limit of the detection frequency band is greater than 100 Hz.

4. The method for locating abnormal sounds in a sphere-linked acoustic system according to claim 1, characterized in that, Step 3 utilizes array spatial information and conventional beamforming technology to locate the sound source, calculates the sound source direction information, and sends it to the monitoring PTZ camera.

5. The method for locating abnormal sounds in a sphere-linked acoustic system according to claim 4, characterized in that, The sound source localization steps are as follows: sound source localization is performed by beamforming; N sound sensors form a receiving array; for each received signal x... i (t) Select the set weighting vector w i (θ) is weighted and summed to obtain the output y(t,θ) of the array; assuming the target signal is a narrowband signal with a center frequency of f, then the input for beamforming... The vector of sampled data and complex weighting coefficients for all array elements is represented as x(t) = [x1(t)x2(t)...x...]. N (t)] T and w(θ)=[w1(θ)w(θ)…w N (θ)] T The beamforming output, expressed in the form of a vector dot product, is y(t,θ)=w H (θ)x(t)=x H (t)w(θ), where the superscript * denotes the complex conjugate operator, the superscript T denotes the transpose of a vector or matrix, and the superscript H denotes the complex conjugate transpose of a vector or matrix; the power spectrum of the beamforming output is expressed as P(θ)=E[|y(t,θ)| 2 ] = w H (θ)Rw(θ); Theoretically, the deflection angle of the sound source relative to the normal of the center of the planar array is (0°, 0°). Using a 3000Hz single-frequency signal as the sound source, a simulation of conventional planar beamforming of a 16-element array is performed under the condition of a receiving signal-to-noise ratio of 0dB.

6. The method for locating abnormal sounds in a sphere-linked acoustic system according to claim 1, characterized in that, In step 4, after the monitoring PTZ camera is started, zero-point calibration is performed to match the 0° position of the monitoring PTZ camera with the 0° position of the air sonar; then, direction calibration is performed to ensure that the rotation direction of the verification PTZ camera is consistent with the direction marked by the sonar. The coordinate information is converted into the rotation angle of the monitoring PTZ camera, which guides the camera to turn in the direction of the abnormal sound. The monitoring PTZ camera continuously captures images of the abnormal sound and stores them together with the alarm recording as an alarm log.

7. The method for locating abnormal sounds in a sphere-linked acoustic system according to claim 6, characterized in that, Step 4 involves continuously capturing images at 1-second intervals and saving the audio data five seconds before and after the alarm as alarm recordings, which are then stored together as an alarm log.

Images