A method and system for speaker source localization based on geometrically constrained global coherence field and visual guidance

Abstract

The present application relates to a kind of speaker sound source positioning method and system based on geometric constraint global coherence field and visual guidance.The method comprises: using camera geometric model to extract geometric constraint global coherence field clue, to represent the spatial characteristics of sound source signal;According to the speaker location annotation in the picture sample of the video frame synchronized with audio signal, generate visual guidance binary pseudo label;Adopt convolutional neural network combined with fully connected network to constitute acoustic network, acoustic network uses geometric constraint global coherence field clue to generate acoustic position atlas, calculates the mean square error of acoustic position atlas and generated pseudo label to train acoustic network;Adopt the acoustic position atlas generated by trained acoustic network, determine the position coordinates of speaker sound source according to the peak position in acoustic position atlas.The present application utilizes the complementarity and consistency between visual mode and auditory mode, enhances the learning ability of acoustic network for sound source position, and can realize accurate sound source positioning.

Description

Technical Field

[0001] This invention belongs to the field of audio and video signal processing, specifically relating to a speaker sound source localization method and system based on geometrically constrained global coherent field and visual guidance. Background Technology

[0002] As a foundation for human-computer interaction, sound source localization plays a crucial role in intelligent human-computer systems, serving as a fundamental step in enabling robots to interact with humans and the environment. Sound source localization refers to locating the relative position or orientation of a sound source using acoustic sensors mounted on or external to the robot. After obtaining the sound source's location or orientation information through sound source localization, the robot can feed this information to higher processing units to achieve further functional requirements. Speaker sound source localization methods play an important role in various fields such as video conferencing, mobile robots, intelligent monitoring, and intelligent shopping guides. However, single-modal speaker sound source localization methods are limited by the modality itself; the auditory modality is affected by environmental acoustic noise, room reverberation, and other factors. Therefore, multimodal methods can be employed, leveraging the complementarity between visual and auditory modalities to improve the accuracy and robustness of speaker sound source localization.

[0003] Traditional sound source localization methods mainly include Time Difference of Arrival (TDA), Controlled Beamforming (CBBE), and High-Resolution Spectral Estimation (HRSE). TDA-based methods utilize the relative time difference between the arrival times of sound source signals at different microphones for localization. While computationally simple, the error from time delay estimation accumulates in the location estimation stage, resulting in a suboptimal estimate of the sound source's location. CBBE can use spatial information for spatial filtering, linearly attenuating interference from unwanted directions or other non-stationary noise. However, CBBE relies on prior information about the sound source and noise signals, which is often difficult to obtain in practice. HRSE-based methods perform eigenvalue decomposition on the covariance matrix of the microphone signal, estimate the spatial spectrum using the orthogonality of the signal and noise subspaces, and finally determine the spatial location of the sound source by searching for the extrema of the spatial spectrum. HRSE is a super-resolution estimation technique whose spatial resolution is not constrained by the microphone signal sampling rate and breaks the Rayleigh limit, achieving arbitrary accuracy under certain conditions. However, the positioning accuracy of this method is closely related to the precision of spatial partitioning. Performing a fine search of the entire positioning candidate space will increase the algorithm complexity.

[0004] With the development of deep learning technology, supervised learning-based sound source localization methods have attracted increasing attention from researchers. Compared with traditional unsupervised methods, deep learning-based sound source localization methods are data-driven, thus adapting well to the acoustic conditions present in the training data. Furthermore, robots' perceptual modalities are multidimensional, and the sound source information provided by vision and hearing is complementary to some extent. Therefore, visual information can be used as a supervisory signal to train the network to learn more accurate sound source location cues. Summary of the Invention

[0005] This invention addresses the problem of speaker source localization for moving speakers in complex scenarios by proposing a speaker source localization method and system based on geometrically constrained global coherence field and visual guidance.

[0006] The technical solution adopted in this invention is as follows:

[0007] A speaker sound source localization method based on geometrically constrained global coherence field and visual guidance includes the following steps:

[0008] A camera geometric model is used to extract geometrically constrained global coherence field (GCF) cues to represent the spatial characteristics of the sound source signal;

[0009] Based on the speaker position annotations in the image samples of video frames synchronized with the audio signal, generate visually guided binary pseudo-labels;

[0010] An acoustic network is constructed by combining a convolutional neural network with a fully connected network. The acoustic network generates an acoustic location map using geometrically constrained global coherence field cues. The mean squared error (MSE) between the acoustic location map and the generated pseudo-labels is calculated to train the acoustic network.

[0011] An acoustic location map is generated using a trained acoustic network, and the location coordinates of the speaker's sound source are determined based on the peak positions in the acoustic location map.

[0012] Furthermore, the extraction of geometrically constrained global coherent field cues using a camera geometric model mainly includes three steps:

[0013] a) Construct a camera geometric model based on the camera's intrinsic and extrinsic parameters;

[0014] b) Set sampling points in the image plane, and use the camera geometry model to project the two-dimensional sampling points into three-dimensional space. Based on the geometry of the room where the speaker is located, remove the three-dimensional sampling points outside the room.

[0015] c) Calculate global coherent field clues based on three-dimensional sampling points.

[0016] Furthermore, the step of generating visually guided binary pseudo-labels based on speaker position annotations in image samples of video frames synchronized with the audio signal includes: generating a two-dimensional Gaussian distribution based on the parameters of the speaker's face annotation box in the image sample, and further generating a binary pseudo-label map through a step function.

[0017] Furthermore, the step of generating an acoustic location map using a trained acoustic network and determining the speaker's sound source location coordinates based on the peak positions in the acoustic location map includes: first, adjusting the size of the acoustic location map according to the step size of the acoustic network; then, filling the acoustic location map with zero values ​​to make the acoustic location map the same size as the input image; and finally, deriving that the location of the maximum value in the acoustic location map is the estimated speaker's sound source coordinates.

[0018] A speaker sound source localization system based on geometrically constrained global coherence field and visual guidance, comprising:

[0019] The global coherence field cue extraction module is used to extract geometrically constrained global coherence field cues using a camera geometric model to represent the spatial characteristics of the sound source signal;

[0020] The pseudo-label generation module is used to generate visually guided binary pseudo-labels based on the speaker position annotations in the image samples of video frames synchronized with the audio signal.

[0021] The acoustic network training module is used to construct an acoustic network by combining a convolutional neural network with a fully connected network. The acoustic network uses geometrically constrained global coherent field cues to generate an acoustic location map, and calculates the mean square error between the acoustic location map and the generated pseudo-labels to train the acoustic network.

[0022] The sound source localization module is used to generate an acoustic location map using a trained acoustic network, and to determine the location coordinates of the speaker's sound source based on the peak positions in the acoustic location map.

[0023] This invention incorporates speaker location annotations from camera models and image samples into acoustic cue extraction and acoustic network training. Leveraging the complementarity and consistency between visual and auditory modalities, it enhances the acoustic network's ability to learn sound source locations. By introducing a camera model, acoustic cues in three-dimensional space are projected onto a localization space on an image plane consistent with the visual modality, allowing audiovisual information to interact within a unified representation space. Furthermore, by introducing visually guided binarized pseudo-labels as supervisory signals for the network, a correlation is established between acoustic cues and visual information about the sound source, prompting the network to generate a more accurate sound source location map, thereby achieving precise sound source localization. Attached Figure Description

[0024] Figure 1 The overall flowchart of the method of the present invention. Detailed Implementation

[0025] To describe the method of the present invention more clearly and completely, the present invention will be further explained below with reference to specific embodiments and accompanying drawings.

[0026] refer to Figure 1 This is a flowchart of the method of the present invention, which mainly includes the following steps:

[0027] Step 1: Extracting geometrically constrained global coherent field acoustic cues

[0028] The extraction process of geometrically constrained global coherent field clues mainly consists of three steps:

[0029] The first step is to construct a camera geometric model based on the camera's intrinsic and extrinsic parameters. This model provides a projection function between the three-dimensional world coordinate system and the two-dimensional image coordinate system captured by the camera.

[0030] The second step involves sampling on the image plane to obtain a series of two-dimensional sampling points. Using the inverse process of the projection function in the camera geometry model, these two-dimensional sampling points are projected onto three-dimensional space. Based on the geometry of the speaker's room, three-dimensional sampling points outside the room are removed.

[0031] The third step, for the multi-channel audio signals from the microphone array, is to first calculate their phase-transform weighted generalized cross-correlation function (GCC-PHAT). This is done for the discrete-time signals s acquired by microphones m1 and m2. m1 (n) and s m1 (n), where n represents the number of audio signal sampling points. The calculation method for GCC-PHAT is as follows:

[0032]

[0033] in, Let F[] denote the phase transform weighted generalized cross-correlation function GCC-PHAT, τ represent the time difference, F[] denotes the discrete Fourier transform, and * is the conjugate complex operator. Theoretically, the GCC-PHAT function at microphones m1 and m2, centered at time t, exhibits a distinct peak, with the time delay corresponding to the peak equal to the actual time difference of arrival. The global coherence coefficient is defined as the average value of the GCC-PHAT for each pair of microphones.

[0034]

[0035] Where p represents the 3D sampling point obtained in the second step, r(p) represents the global coherence field coefficient at point p, R is the room range, and M represents the number of microphone pairs. The global coherence field coefficients at all 3D sampling points are calculated and arranged into a matrix according to the location distribution of the sampling points; this matrix represents the global coherence field cues.

[0036] Step 2: Generate visually guided binary pseudo-labels

[0037] For video frames synchronized with the audio signal, label the speaker's face position (u, v, h). f ,w f ), where u represents the horizontal axis coordinate, v represents the vertical axis coordinate, and h f Indicates the vertical height of the face, w f This represents the horizontal width of the face; a two-dimensional Gaussian distribution and binarized pseudo-labels are generated based on the face bounding box, i.e.:

[0038]

[0039] Where (x, y) are the coordinates on the pseudo-label map. Therefore As the mean, let ∑=diag(h) f / ι,w f / ι) represents the two-dimensional Gaussian distribution of the covariance matrix, ι is a scaling factor used to control the range of the Gaussian distribution, and ε λ It is a step function with a threshold of λ.

[0040] Step 3: Training the acoustic network

[0041] The acoustic network used is composed of a convolutional neural network combined with a fully connected network, containing 6 convolutional layers, 3 fully connected layers, and one average pooling layer. Its network structure and parameters are shown in Table 1. This acoustic network uses the geometrically constrained global coherence field cues from step 1 to generate an acoustic location map. Specifically, the geometrically constrained global coherence field cues from step 1 are input into the acoustic network, and after computation, the acoustic location map is output. The acoustic network uses visually guided pseudo-labels as supervision signals to estimate the acoustic location map of the sound source, thereby deriving the speaker's sound source location coordinates.

[0042] The acoustic location map output by the acoustic network and the pseudo-labels in step 2 are used to calculate the loss using mean squared error (MSE) to train the acoustic network.

[0043] Table 1. Network structure of acoustic networks

[0044] enter layer set up Output (3,400,400) conv1 conv2d(3,96,11,2)maxpool2d(3,2) (96,97,97) (96,97,97) conv2 conv2d(96,256,5,1)maxpool2d(3,2) (256,46.46) (256,46.46) conv3 conv2d(256,384,3,1) (384,44,44) (384,44,44) conv4 conv2d(384,384,3,1) (384,42,42) (384,42,42) conv5 conv2d(384,256,3,1) (256,40,40) (256,40,40) conv6 conv2d(256,256,6,1) (256,35,35) (256,35,35) fc1 fc(256,128) (128,35,35) (128,35,35) fc2 fc(128,128) (128,35,35) (128,35,35) fc3 fc(128,256) (256,35,35) (256,35,35) avgpool avgpool (1,35,35)

[0045] Step 4: Determine the coordinates of the speaker's sound source

[0046] A trained acoustic network is used to generate an acoustic location map. The size of the acoustic location map is adjusted according to the network's step size, and then zero values ​​are padded around the map to make the map the same size as the input image. Finally, the location of the maximum value in the map is derived as the estimated coordinates of the speaker's sound source.

[0047] Another embodiment of the present invention provides a speaker sound source localization system based on geometrically constrained global coherence field and visual guidance, comprising:

[0048] The global coherence field cue extraction module is used to extract geometrically constrained global coherence field cues using a camera geometric model to represent the spatial characteristics of the sound source signal;

[0049] The pseudo-label generation module is used to generate visually guided binary pseudo-labels based on the speaker position annotations in the image samples of video frames synchronized with the audio signal.

[0050] The acoustic network training module is used to construct an acoustic network by combining a convolutional neural network with a fully connected network. The acoustic network uses geometrically constrained global coherent field cues to generate an acoustic location map, and calculates the mean square error between the acoustic location map and the generated pseudo-labels to train the acoustic network.

[0051] The sound source localization module is used to generate an acoustic location map using a trained acoustic network, and to determine the location coordinates of the speaker's sound source based on the peak positions in the acoustic location map.

[0052] For the specific implementation process of each module, please refer to the description of the method of the present invention above.

[0053] Another embodiment of the present invention provides a computer device (computer, server, smartphone, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the method of the present invention.

[0054] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) storing a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0055] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention without departing from the spirit and scope of the present invention. The scope of protection of the present invention should be determined by the claims.

Claims

1. A speaker sound source localization method based on geometrically constrained global coherence field and visual guidance, comprising the following steps: A camera geometric model is used to extract geometrically constrained global coherent field cues to represent the spatial characteristics of the sound source signal; Based on the speaker position annotations in the image samples of video frames synchronized with the audio signal, generate visually guided binary pseudo-labels; An acoustic network is constructed by combining a convolutional neural network with a fully connected network. The acoustic network generates an acoustic location map using geometrically constrained global coherent field cues. The mean square error between the acoustic location map and the generated pseudo-labels is calculated to train the acoustic network. An acoustic location map is generated using a trained acoustic network, and the location coordinates of the speaker's sound source are determined based on the peak positions in the acoustic location map.

2. The method as described in claim 1, characterized in that, The extraction of geometrically constrained global coherent field cues using a camera geometric model includes: Construct a camera geometric model based on the camera's intrinsic and extrinsic parameters; Sampling points are set in the image plane, and the two-dimensional sampling points are projected into three-dimensional space using the camera geometry model. Based on the geometry of the room where the speaker is located, the three-dimensional sampling points outside the room are removed. Global coherent field cues are calculated based on three-dimensional sampling points.

3. The method as described in claim 2, characterized in that, The calculation method for the global coherent field clues based on three-dimensional sampling points is as follows: Where r(p) represents the global coherence field coefficient at point p, and R is the room extent. The phase-transform weighted generalized cross-correlation function is represented, and M represents the number of microphone pairs; the global coherence field coefficients at all three-dimensional sampling points are calculated to form a global coherence field cue.

4. The method as described in claim 1, characterized in that, The step of generating visually guided binary pseudo-labels based on speaker position annotations in image samples of video frames synchronized with audio signals includes: generating a two-dimensional Gaussian distribution based on the parameters of the speaker's face annotation box in the image sample, and then generating a binary pseudo-label map through a step function.

5. The method as described in claim 4, characterized in that, The pseudo-label image is calculated using the following formula: Where (x, y) are the coordinates on the pseudo-label map. It is a two-dimensional Gaussian distribution with mean μ and covariance matrix ∑, ε λ It is a step function with a threshold of λ.

6. The method as described in claim 1, characterized in that, The acoustic network consists of a convolutional neural network combined with a fully connected network, and uses visually guided binarized pseudo-labels as supervision signals to estimate the acoustic location map of the sound source.

7. The method as described in claim 1, characterized in that, The process of generating an acoustic location map using a trained acoustic network and determining the speaker's sound source location coordinates based on the peak positions in the acoustic location map includes: first, adjusting the size of the acoustic location map according to the step size of the acoustic network; then, filling the acoustic location map with zero values ​​to make the acoustic location map the same size as the input image; and finally, deriving that the location of the maximum value in the acoustic location map is the estimated speaker's sound source coordinates.

8. A speaker sound source localization system based on geometrically constrained global coherence field and visual guidance, characterized in that, include: The global coherence field cue extraction module is used to extract geometrically constrained global coherence field cues using a camera geometric model to represent the spatial characteristics of the sound source signal; The pseudo-label generation module is used to generate visually guided binary pseudo-labels based on the speaker position annotations in the image samples of video frames synchronized with the audio signal. The acoustic network training module is used to construct an acoustic network by combining a convolutional neural network with a fully connected network. The acoustic network generates an acoustic location map using geometrically constrained global coherent field cues, and the mean square error between the acoustic location map and the generated pseudo-labels is calculated to train the acoustic network. The sound source localization module is used to generate an acoustic location map using a trained acoustic network, and to determine the location coordinates of the speaker's sound source based on the peak positions in the acoustic location map.

9. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method according to any one of claims 1 to 7.

Images