Audio-visual speaker tracking method based on semantic-spatial feature fusion

Abstract

The invention belongs to the technical field of multi-mode perception, and particularly relates to an audiovisual speaker tracking method based on semantic-spatial feature fusion, which comprises the following steps of: acquiring original audio and video signals, and extracting semantic feature codes in audiovisual signals by adopting an audiovisual fusion keyword recognition network; respectively processing the image frame and the acoustic signal by adopting a double-flow network structure comprising a visual branch and an auditory branch; a cross attention module is adopted to realize information interaction and fusion between two different semantic-spatial feature sequences. According to the method, the correlation and complementarity between different level features and different modal features are mined by adopting a cross attention mechanism, and information interaction between different information sources is promoted. A semantic-spatial feature fusion mechanism can adaptively pay attention to valuable information, multi-level and cross-modal space-time consistency feature representation is learned, the expression ability of audio-visual features is further improved, and therefore more accurate tracking is achieved.

Description

Technical Field

[0001] The present invention belongs to the technical field of multimodal perception, and in particular relates to an audiovisual speaker tracking method based on semantic-spatial feature fusion. Background Art

[0002] In recent years, with the development of artificial intelligence and multimodal learning technology, audio-visual speaker tracking, as a basic task in multimodal perception and intelligent systems, has become a research topic that has attracted much attention. Audio-visual speaker tracking aims to accurately track the speaker's position by combining visual and auditory information, and has broad application prospects in video conferencing, intelligent monitoring, human-computer interaction and other fields.

[0003] Currently, researchers have proposed a variety of speaker tracking methods, which can be roughly divided into methods based on visual features, methods based on auditory features, and methods based on audio-visual fusion. Methods based on visual features mainly use image data captured by cameras to detect and track the speaker's facial features or other salient features to achieve target tracking. These methods perform well in environments with sufficient lighting and clear vision, but their performance will drop significantly in the presence of occlusion, lighting changes, or complex backgrounds. Methods based on auditory features mainly use audio data captured by microphone arrays to locate and track speakers by analyzing the time-frequency features and spatial clues in the sound signal. These methods work better in conditions with low noise and less environmental echo, but in noisy environments or when there are multiple sound sources, the stability and accuracy of acoustic features will be greatly affected. Audio-visual fusion methods attempt to combine visual and auditory information to make up for the shortcomings of single modal features. The robustness and accuracy of speaker tracking can be improved by fusing multimodal data.

[0004] However, existing audiovisual speaker tracking methods mainly focus on utilizing spatial position information in visual and auditory data, such as building an appearance model of the target, performing similarity search in various regions of the tracking space, or extracting spatial clues of sound sources in multi-channel signals using spectrum analysis and time-frequency analysis. These methods have high requirements on the quality and stability of visual features and acoustic clues. In the presence of occlusion, deformation, complex background in the image, noise, reverberation, and multiple sound sources in the environment, the spatial position information may fail or be insufficient to distinguish the target from the interferers. In addition, existing trackers usually rely on features at a single level, ignoring the complementarity between features at different levels. High-level features have strong semantic information and robustness, but lack detail information and spatial accuracy; low-level features have high spatial resolution and detail information, but lack semantic information and anti-interference ability. Considering that multimodal signals contain rich semantic features, the advantages of features at different levels can be fully utilized to provide more contextual information for tracking targets. Summary of the invention

[0005] In view of the above technical problems, the present invention provides an audio-visual speaker tracking method based on semantic-spatial feature fusion, which improves the expressiveness of audio-visual features and achieves more accurate sound source tracking.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] An audio-visual speaker tracking method based on semantic-spatial feature fusion comprises the following steps:

[0008] S1, obtain the original audio and video signal, use the audio-visual fusion keyword recognition network to extract the semantic feature coding in the audio-visual signal;

[0009] S2, using a two-stream network structure including a visual branch and an auditory branch to process image frames and acoustic signals respectively;

[0010] S3. Use the cross-attention module to realize the information interaction and fusion between two different sequences of semantic-spatial features.

[0011] The method for extracting semantic feature coding from the audio-visual signal in S1 is:

[0012] S11. Convert the original audio and video signals into high-dimensional feature representation F a With F v ;

[0013] S12. Use a Transformer-based encoder to obtain semantic features related to keywords.

[0014] The processing process of the semantic feature encoder in S12 is expressed as:

[0015] Encoder(X) = X att +FFN(X att )

[0016] X att =X+MHA(Q,K,V)

[0017] FFN(X att )=max(0,X att W 1 +b 1 )W 2 +b 2

[0018] Where X is the input of the encoder, that is, [CLS a ; F a ] or [CLS v ; F v] plus position encoding; FFN(·) is a fully connected feedforward network used to enhance the model fitting ability, including two linear transformation layers with a ReLU activation function in between; W and b represent the weight matrix and basis vector respectively, and the subscripts represent different layers; CLS a and CLS v is the CLS tag for classification.

[0019] The method of using a dual-stream network structure including a visual branch and an auditory branch in S2 to process image frames and acoustic signals respectively is:

[0020] S21, input a pair of images into the visual space feature extraction network, including the reference template I tpl and search area image I s ; Both are input into a fully convolutional network with shared weights to extract features;

[0021] S22, performing a cross-correlation operation based on a convolution operator to obtain a response map; the operation process of the visual network is defined as follows:

[0022] f v =Conv v (I tpl )*Conv v (I s )

[0023] Among them, Conv v These are two fully convolutional networks with the same structure and shared weights. * indicates the cross-correlation operation.

[0024] S23, projecting acoustic cues onto the image plane using a camera model;

[0025] S24, then use a fully convolutional network structure similar to the visual network to embed the sound signal into a consistent positioning space containing the position context; the operation process of the auditory network is defined as:

[0026] f a =Conv a (R Ω )

[0027] Among them, R Ω are manually extracted acoustic cues of stGCF.

[0028] The method of using the cross attention module in S3 to realize the information interaction and fusion between two different sequences of semantic-spatial features is:

[0029] S31. Take the features of one modality as Q and the features of the other modality as K and V for attention calculation;

[0030] S32 and use residual multi-head attention to integrate information from different sequences.

[0031] The method of using residual multi-head attention in S32 to integrate information from different sequences is:

[0032] The CA mechanism is defined as follows:

[0033]

[0034] Among them, Q A It's A+E A The linear transformation of Q B It's B+E B The linear transformation of E A and E B It is used to supplement the position coding of spatial position information. The definitions of K and V are similar. LN(·) represents layer normalization.

[0035] Then, a two-step fusion is used to enhance the feature distinguishability of different tracking targets in multi-person scenes, including intra-modality and inter-modality stages. First, semantic-spatial feature fusion is performed within each modality, and then audio-visual fusion is performed between different modalities. The fusion of the first stage can be formally expressed as:

[0036] f sv =f v +MHA(Q sv ,K v ,V v )

[0037] f sa ′=f a +MHA(Q sa ,K a ,V a )

[0038] Among them, Q sv and Q sa are the linear transformations of the auditory and visual semantic feature encodings output by the encoder of the AVKS network, and K v and V v is the visual spatial feature f v Linear transformation, K a and V a is the auditory spatial feature f a The linear transformation of ; the cross-modal fusion process in the second stage is expressed as:

[0039] f sa =f sa ′+MHA(Q sv ′,K sa ′,V sa ′)

[0040] f av =CA(f sa ,f sv )

[0041] Among them, K sa ′ and V sa ′ is f sa ′, Q sv ′ is another linear transformation of visual semantic feature encoding.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] The visual semantic feature encoding of the present invention reflects the speaking state of the target, and the attention map in the auditory semantic feature encoder reflects the position of the keyword in the input sequence. The present invention adopts a cross-attention mechanism to explore the correlation and complementarity between features of different levels and features of different modalities, and promote information interaction between different information sources. The semantic-spatial feature fusion mechanism can adaptively focus on valuable information, learn multi-level and cross-modal spatiotemporal consistency feature representations, further improve the expressiveness of audio-visual features, and thus achieve more accurate tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0045] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0046] Figure 1 It is a schematic diagram of the network model of the present invention.

[0047] Figure 2 It is a schematic diagram of the AVKS network structure of the present invention.

[0048] Figure 3 Schematic diagram of the four fusion methods designed for the present experiments.

[0049] Figure 4Schematic diagram for visualizing the semantic-spatial features of the visual branch and the auditory branch of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. These descriptions are only to further illustrate the features and advantages of the present invention, rather than to limit the claims of the present invention. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0051] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0052] The present invention provides an Audio-Visual Tracker (AVSS-Tracker) based on semantic-spatial feature fusion. Through an audio-visual semantic feature encoding method based on an audio-visual keyword recognition network that can extract semantic information related to a target from an audio-visual signal, and a two-stage semantic-spatial feature fusion mechanism, a cross-attention mechanism is used to mine the correlation and complementarity of features at different levels and modalities, which significantly improves the feature expression ability and tracking accuracy. In the actual application process, such as Figure 1 As shown, the following steps are included:

[0053] S1. Get the original audio and video signal, use the audio-visual fusion keyword spotting network (Audio-Visual Keyword Spotting, AVKS) to extract the semantic feature coding in the audio-visual signal. The original audio and video signal is converted into a high-dimensional feature representation F a With F v , the audio-visual features are further encoded into a higher-level semantic representation, using a Transformer-based encoder to obtain semantic features related to keywords. The processing of the semantic feature encoder is shown as follows:

[0054] Encoder(X) = X att +FFN(X att )

[0055] X att =X+MHA(Q,K,V)

[0056] FFN(X att )=max(0,X att W 1 +b1 )W 2 +b 2

[0057] Where X is the input of the encoder, that is, [CLS a ; F a ] or [CLS v ; F v ] plus position encoding. FFN(·) is a fully connected feedforward network used to enhance the model fitting ability, including two linear transformation layers with a ReLU activation function in between. W and b represent the weight matrix and basis vector respectively, and the subscripts represent different layers.

[0058] CLS a and CLS v is the CLS tag for classification.

[0059] S2. A two-stream network structure consisting of a visual branch and an auditory branch is adopted to process image frames and acoustic signals respectively.

[0060] like Figure 2 , Figure 4 As shown, the input of the visual space feature extraction network is a pair of images, including the reference template I tpl and search area image I s . Both are input into a fully convolutional network with shared weights to extract features. Then a cross-correlation operation based on the convolution operator is performed to obtain the response map. The operation process of the visual network is defined as follows:

[0061] f v =Conv v (I tpl )*Conv v (I s )

[0062] The response value represents the probability of the template at each position in the search area, where Conv v These are two fully convolutional networks with the same structure and shared weights. * indicates the cross-correlation operation.

[0063] Auditory spatial feature extraction is based on the stGCF acoustic spectrogram extraction algorithm, using the camera model to project acoustic cues onto the image plane. Then, a fully convolutional network structure similar to the visual network is used to embed the sound signal into a consistent positioning space containing position context. The operation process of the auditory network is defined as follows:

[0064] f a =Conv a (R Ω )

[0065] Among them, R Ωare manually extracted acoustic cues of stGCF.

[0066] S3. The Cross Attention (CA) module is used to realize the information interaction and fusion between two different sequences. The features of one modality are used as Q, and the features of the other modality are used as K and V for attention calculation. And the residual form of Multi-Head Attention (MHA) is used to integrate the information from different sequences. The definition of the CA mechanism is as follows:

[0067]

[0068] Among them, Q A It's A+E A The linear transformation of Q B It's B+E B The linear transformation of E A and E B It is a positional encoding used to supplement the spatial position information. The definitions of K and V are similar. LN(·) represents layer normalization.

[0069] Then, a two-step fusion is used to enhance the feature distinguishability of different tracking targets in multi-person scenes, including intra-modality and inter-modality stages. First, semantic-spatial feature fusion is performed within each modality, and then audio-visual fusion is performed between different modalities. The first stage of fusion can be formally expressed as:

[0070] f sv =f v +MHA(Q sv ,K v ,V v )

[0071] f sa ′=f a +MHA(Q sa ,K a ,V a )

[0072] Among them, Q sv and Q sa are the linear transformations of the auditory and visual semantic feature encodings output by the encoder of the AVKS network, and K v and V v is the visual spatial feature f v Linear transformation, K a and V a is the auditory spatial feature f a The cross-modal fusion process in the second stage is expressed as:

[0073] f sa =fsa ′+MHA(Q sv ′,K sa ′,V sa ′)

[0074] f av =CA(f sa ,f sv )

[0075] Among them, K sa ′ and V sa ′ is f sa ′, Q sv ′ is another linear transformation of visual semantic feature encoding.

[0076] Dataset

[0077] AV16.3 Dataset: This dataset is a real indoor multi-speaker audio-visual corpus specifically designed to test audio-only, video-only, and audio-visual fusion positioning and tracking algorithms. The dataset contains 43 audio-visual sequences with a duration of 14 seconds to 9 minutes, containing 1 to 3 mobile speakers, recorded using a distributed sensor platform. The audio data is recorded at a sampling rate of 16kHz by two 8-microphone uniform circular arrays with a radius of 10 cm placed on a conference table. The video is captured by 3 digital cameras at a sampling rate of 25Hz, and each frame is a color image of 360×288 pixels. The experiments in this chapter use data from one camera and one microphone and are tested on 9 sequences in three single-speaker scenes (seq08, 11, 12) of AV16.3-SOT.

[0078] CAV3D dataset: This dataset is a 3D speaker tracking audio-visual corpus collected by a co-located sensor platform. The sensing platform consists of a camera and an 8-element circular microphone array with a diameter of 20 cm, recording 8-channel audio signals at 96kHz (24 bits) and 768×1024 pixel video signals at 15 frames per second (fps). In addition, 4 corner cameras are configured for 3D trajectory calibration. The experiments used two subsets: CAV3D-SOT2 (6 scenes, one speaker and one silent participant) and CAV3D-MOT (5 scenes, multiple simultaneous speakers).

[0079] Experimental setup

[0080] The proposed AVKS and AVSS-Tracker use a phased training strategy. First, the classifiers of the visual branch and the auditory branch of AVKS are pre-trained, and then the unimodal models are combined to train the audio-visual fusion model. The training uses cross entropy as the loss function and is performed for 35 cycles with an initial learning rate of 0.01. During testing, 5 seconds of audio-visual signals are captured as the input of the semantic feature encoding network. The image samples for training AVSS-Tracker are 120×120 image patches captured from AV16.3 single-person sequences seq01, 02, and 03. The image samples are randomly cropped around the ground-truth bounding box and completely contain the ground-truth box. The audio samples are stGCF atlases extracted from the corresponding sampling area. A total of 44,985 audio-visual sample pairs are collected. The parameters of the semantic feature encoding part are fixed to train AVSS-Tracker, and the SGD optimizer is used for 50 cycles, with a batch size of 16 and a fixed learning rate of 0.0001. All experiments are based on the PyTorch environment and performed on an NVIDIA RTX 4090Ti graphics card.

[0081] This task involves multiple target tracking, so in addition to 2D MAE (unit: pixel) and positioning accuracy (ACC, unit: %), MOT (Multiple Object Tracking) metrics are used as evaluation indicators, including MOTA (MOTAccuracy, unit: %) and MOTP (MOT Precision, unit: pixel).

[0082] Effectiveness evaluation of semantic-spatial feature fusion

[0083] In order to evaluate the effectiveness of the semantic-spatial feature fusion module, different fusion strategies are compared on two sequences, CAV3D-SOT2 and AV16.3-SOT. Four fusion methods are designed in the experiment, such as Figure 3 As shown in the figure. Method 1 does not add semantic features, but only uses the cross-attention mechanism to fuse audiovisual spatial features. Method 2 directly adds the audiovisual features element by element after performing semantic-spatial feature fusion within the modality, lacking cross-modal information interaction. Method 3 adds a cross-modal audiovisual cross-attention module on the basis of Method 2. Method 4 adopts a two-stage fusion strategy, which adds the fusion of visual semantic features and auditory spatial features on the basis of Method 3. The new network was trained according to different fusion methods, and the tracking performance is shown in Table 2. It can be seen that the method of using both spatial-feature fusion and cross-modal fusion is better than the two one-stage fusion methods, which proves the effectiveness of semantic features and the effectiveness of the cross-modal attention mechanism.

[0084] Only the preferred embodiments of the present invention are described in detail above, but the present invention is not limited to the above embodiments. Various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention, and various changes should be included in the protection scope of the present invention.

Claims

1. An audio-visual speaker tracking method based on semantic-spatial feature fusion, characterized in that: The following steps are involved: S1, obtain the original audio and video signal, use the audio-visual fusion keyword recognition network to extract the semantic feature coding in the audio-visual signal; S2, using a two-stream network structure including a visual branch and an auditory branch to process image frames and acoustic signals respectively; S3. Use the cross-attention module to realize the information interaction and fusion between two different sequences of semantic-spatial features.

2. The audio-visual speaker tracking method based on semantic-spatial feature fusion according to claim 1, characterized in that: The method for extracting semantic feature coding from the audio-visual signal in S1 is: S11. Convert the original audio and video signals into high-dimensional feature representation F a With F v ; S12. Use a Transformer-based encoder to obtain semantic features related to keywords.

3. The audio-visual speaker tracking method based on semantic-spatial feature fusion according to claim 1, characterized in that: The processing process of the semantic feature encoder in S12 is expressed as: Encoder(X)=X att +FFN(X att ) X att =X+MHA(Q,K,V) FFN(X att )=max(0,X att W1+b1)W2+b2 Where X is the input of the encoder, that is, [CLS a ; F a ] or [CLS v ; F v ] plus position encoding; FFN(·) is a fully connected feedforward network used to enhance the model fitting ability, including two linear transformation layers with a ReLU activation function in between; W and b represent the weight matrix and basis vector respectively, and the subscripts represent different layers; CLS a and CLS v is the CLS tag for classification.

4. The audio-visual speaker tracking method based on semantic-spatial feature fusion according to claim 1, characterized in that: The method of using a dual-stream network structure including a visual branch and an auditory branch in S2 to process image frames and acoustic signals respectively is: S21, input a pair of images into the visual space feature extraction network, including the reference template I tpl and search area image I s ; Both are input into a fully convolutional network with shared weights to extract features; S22, performing a cross-correlation operation based on a convolution operator to obtain a response map; the operation process of the visual network is defined as follows: f v =Conv v (I tpl )*Conv v (I s ) Among them, Conv v These are two fully convolutional networks with the same structure and shared weights. * indicates the cross-correlation operation. S23, projecting acoustic cues onto the image plane using a camera model; S24, then use a fully convolutional network structure similar to the visual network to embed the sound signal into a consistent positioning space containing the position context; the operation process of the auditory network is defined as: f a =Conv a (R Ω ) Among them, R Ω are manually extracted acoustic cues of stGCF.

5. The audio-visual speaker tracking method based on semantic-spatial feature fusion according to claim 1, characterized in that: The method of using the cross attention module in S3 to realize the information interaction and fusion between two different sequences of semantic-spatial features is: S31. Take the features of one modality as Q and the features of the other modality as K and V for attention calculation; S32 and use residual multi-head attention to integrate information from different sequences.

6. The audio-visual speaker tracking method based on semantic-spatial feature fusion according to claim 5, characterized in that: The method of using residual multi-head attention in S32 to integrate information from different sequences is: The CA mechanism is defined as follows: Among them, Q A It's A+E A The linear transformation of Q B It's B+E B The linear transformation of E A and E B It is used to supplement the position coding of spatial position information. The definitions of K and V are similar. LN(·) represents layer normalization. Then, a two-step fusion is used to enhance the feature distinguishability of different tracking targets in multi-person scenes, including intra-modality and inter-modality stages. First, semantic-spatial feature fusion is performed within each modality, and then audio-visual fusion is performed between different modalities. The fusion of the first stage can be formally expressed as: f sv =f v +MHA(Q sv ,K v ,V v ) f sa′ =f a +MHA(Q sa ,K a ,V a ) Among them, Q sv and Q sa are the linear transformations of the auditory and visual semantic feature encodings output by the encoder of the AVKS network, and K v and V v is the visual spatial feature f v Linear transformation, K a and V a is the auditory spatial feature f a The linear transformation of ; the cross-modal fusion process in the second stage is expressed as: f sa =f sa′ +MHA(Q sv′ ,K sa′ ,V sa′ ) f av =CA(f sa ,f sv ) Among them, K sa′ and V sa′ Yes sa′ The linear transformation of Q sv′ It is another linear transformation of visual semantic feature encoding.

Images