Audio and video target tracking method based on global-local information interaction mechanism

Abstract

The invention belongs to the technical field of audio and video target tracking, and particularly relates to an audio and video target tracking method based on a global-local information interaction mechanism, which comprises the following steps of: inputting a global video signal into a visual guidance network, and inputting a full-channel audio signal into an auditory guidance network; inputting the local video signal and the partial channel audio signal into a backbone network; performing global feature reconstruction on the features with local information loss; a heterogeneous signal multi-level feature fusion attention module is introduced, and knowledge of a guide network is transmitted to a backbone network; model parameters are optimized by calculating a total loss function. The generator globally reconstructs features with local information loss in a backbone network, a heterogeneous signal multi-level feature fusion attention module is also provided, and the module dynamically fuses local features and global reconstruction features of heterogeneous signals by using attention weights. Therefore, complementarity and consistency of multi-level characteristics of heterogeneous signals are effectively utilized.

Description

Technical Field

[0001] The invention belongs to the technical field of audio and video target tracking, and in particular relates to an audio and video target tracking method based on a global-local information interaction mechanism. Background Art

[0002] Audio and video target tracking is a key task for intelligent systems to achieve behavioral analysis and human-computer interaction. In order to achieve accurate tracking, researchers are increasingly inclined to use multimodal sensors to capture richer information. In particular, hearing and vision, as the main modalities for humans to understand the surrounding environment and interact, have received widespread attention. Similar to the multimodal perception process of humans, the complementarity of auditory and visual information can provide necessary supplementary clues. In particular, in the speaker tracking task under missing modalities, when the video signal is occluded or the visual sensor fails, the audio signal can serve as a supplementary clue. Similarly, when the audio signal is interfered by noise or the auditory sensor fails, the video signal can provide supplementary information. However, the above method is only applicable to the case where one modality is partially missing or completely missing, and the information of another incomplete modality is supplemented by a high-confidence modality. However, when the information of multiple modalities is partially missing, neither modality can provide reliable observation results, and the fusion effect will drop sharply. Therefore, it is crucial to develop a multimodal tracker that can cope with the incompleteness of multimodal heterogeneous signals and handle occlusion, noise and sensor failure at the same time.

[0003] At present, research on supplementing missing modalities has made great progress. For example, some researchers use the joint attention mechanism of recursive fusion to interact and supplement information between modalities, and some researchers use the existing modal information to supplement the missing modality by sharing specific feature encoders, so that good tracking performance can be maintained when the modality is missing. However, the above methods do not fundamentally complete and reconstruct the missing modal information, but only supplement it with another modal information.

[0004] Existing audio and video target tracking algorithms have proposed a variety of fusion strategies to combine audio and video signals, but have not focused on the multi-level fusion of global and local features in heterogeneous signals. When faced with dynamic environmental changes, they have ignored the enhancement effect of global context information on local features. Summary of the invention

[0005] In view of the technical problem that the above-mentioned existing audio and video target tracking algorithms do not pay attention to the multi-level fusion of global and local features in heterogeneous signals, the present invention provides an audio and video target tracking method based on a global-local information interaction mechanism, which solves the problem of local mode missing in audio and video target tracking tasks.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] A method for tracking audio and video targets based on a global-local information interaction mechanism comprises the following steps:

[0008] S1, obtain audio and video signals, input the global video signal into the visual guidance network, and input the full-channel audio signal into the auditory guidance network to extract global visual and auditory features at different levels;

[0009] S2, inputting the local video signal and some channel audio signals into the backbone network to extract the visual and auditory features with local information missing at different levels;

[0010] S3. Construct a feature reconstruction module based on a generative adversarial network in the backbone network to reconstruct global features of features with local information missing;

[0011] S4, introduces a heterogeneous signal multi-level feature fusion attention module, which integrates features from heterogeneous signals and different levels to achieve complementary and consistent utilization of heterogeneous signal multi-level features;

[0012] S5. Through the global to local knowledge distillation architecture, the knowledge of the guide network is transferred to the backbone network, so that the backbone network can learn global features and improve the robustness to local information loss. The fused heterogeneous signal features are input into the prediction head for target positioning and generate target tracking results.

[0013] S6. Optimize the model parameters by calculating the total loss function. The total loss includes feature alignment loss, generative adversarial loss, logits distillation loss, and position estimation loss.

[0014] The method of guiding the network in S1 to extract visual and auditory features at different levels is as follows: visual guidance network: training the model in the initial offline stage to convert the tracking problem into a similarity measure between a known target and a search area; the visual network adopts a Siamese network structure to calculate a similarity response map between target template features and search area features through a cross-correlation operation;

[0015]

[0016] Where I t is the picture frame currently input to the network, I tem is the reference template, i.e. the user-defined tracking target in the first frame; is a convolutional neural network-based visual encoder with shared weights; * indicates that the convolution operation is used for similarity measurement; the response map V t It reflects the probability of the target appearing at each position in the search image and is used to provide visual positioning clues;

[0017] Auditory Guidance Network: The auditory guidance network consists of an acoustic localization cue extraction module and a codec-based deep feature extraction module, and its input is a multi-channel audio signal from a microphone array. First, the acoustic localization cues based on time and space are extracted, and the audio-visual space mapping is constructed based on the camera model to obtain a sampling grid of potential sound source positions at different depths in three-dimensional space. Then, the GCF map at the sampling grid is calculated through the coherence between audio signals. The depth at which the GCF mapping peak is located is selected to generate a spatial GCF mapping sGCF. Considering the intermittent nature of the speech signal and the continuity of the speaker's movement, the sGCF mapping with the largest peak within a period of time is selected to generate a spatiotemporal GCF mapping stGCF. Subsequently, the extracted acoustic cues learn higher-dimensional and deep-level deep features in the codec-based network. The audio feature A t ' is defined as follows:

[0018]

[0019] Among them, U t is the 16-channel audio signal from the microphone array, R Ω (g) represents the extraction of acoustic localization cues, Denotes the encoder and decoder of the auditory guided network, A t Represents the result of the auditory guided encoder; this hybrid feature extraction method uses both manually designed prior knowledge and the powerful expression ability of neural networks to improve feature quality and positioning effect.

[0020] The backbone network in S2 consists of a lightweight audio-visual encoder, a feature reconstruction module, a heterogeneous signal multi-level feature fusion attention module, and a prediction head.

[0021] The method for extracting visual and auditory features with local information missing at different levels by the backbone network in S2 is:

[0022] S2.1, the local features extracted by the audio-visual encoder are used to complete the features with missing local information through the feature reconstruction module;

[0023] S2.2, the audio-visual local features and the reconstructed global features are input into the fusion attention module for fusion;

[0024] S2.3. Output the target positioning result through the prediction head.

[0025] The feature reconstruction method of the feature reconstruction module of the generative adversarial network in S3 is as follows: a feature reconstruction module is designed in the backbone network and deployed on the visual and auditory branches respectively; first, the generator G is used to receive the missing features and reconstruct the global features, and then the discriminator D is used to distinguish whether these features are reconstructed features or real global features from the guided network; the visual and auditory generators are composed of encoders and decoders, and the discriminator is composed of multiple convolutional layers and fully connected layers; let V s , A s are the visual and auditory features of the input generator, V′, A t ' is the global audio-visual feature to guide the network, and the discriminator is processed as follows:

[0026]

[0027] in: Represents the generated visual reconstruction features and auditory reconstruction features; discriminator D v and D a The output of represents the probability that the input feature is a true feature; by alternately optimizing the generator and the discriminator, the generator gradually learns to reconstruct the missing features and assists the backbone network in processing incomplete observation data.

[0028] The method for multi-level feature fusion of heterogeneous signals in S4 is:

[0029] First, the local features and the global reconstruction features are fused at multiple levels based on attention to obtain the global-local features. make The perceptual weights generated for the global-local attention module are higher in reliable observations and lower in ambiguous observations with visual occlusion and limited hearing. Then, position encoding is added to the global-local fusion features to improve the tracker's ability to capture and utilize spatial structure information by combining spatial position information. Finally, the feature fusion of audiovisual heterogeneous signals is performed through the heterogeneous signal cross-attention mechanism. The calculation process of the multi-level feature fusion attention module is as follows:

[0030]

[0031] Among them, MMCA stands for the heterogeneous signal cross attention mechanism, which consists of two multi-head attention modules, one modal signal as qurey and the other modal signal as key and value. By calculating the correlation between heterogeneous signals, the information of different heterogeneous signals can be effectively interacted and integrated; F av is the fused audio-visual feature; Pos represents the position code; the proposed heterogeneous signal multi-level feature fusion attention module balances the local and global information of audio-visual heterogeneous signals through two-step fusion.

[0032] The method of the global to local knowledge distillation strategy in S5 is: the global to local knowledge distillation strategy includes feature alignment distillation, generative adversarial distillation and logits distillation;

[0033] Feature alignment distillation: The output of the cross-correlation operation of the backbone network visual branch is represented as V s , specifically Among them I s It is the image frame with partial missing input by the backbone network. represents the visual encoder of the backbone network; the output of the backbone network auditory encoder is represented as A s , specifically Among them U s is the local channel audio signal from the microphone array, R Ω′ (g) represents the acoustic localization clues of the backbone network. represents the auditory encoder of the backbone network; the feature alignment loss is based on the MSE loss and is divided into two parts, visual and auditory, expressed as:

[0034]

[0035] Where N is the dimension of the feature vector, represents the MSE loss operator; feature alignment distillation improves the consistency of feature representation in the guided backbone network, ensuring that the backbone network can capture the main patterns and information of heterogeneous signals even if the input signal is incomplete;

[0036] Generative adversarial distillation: Generative adversarial distillation is based on the idea of ​​generative adversarial networks (GANs). Through adversarial training, the reconstructed features generated by the backbone network are as similar as possible to the real features extracted by the guide network, thereby improving the backbone network's ability to capture global information. Generative adversarial loss includes two parts: visual and auditory. The overall objective function is expressed as:

[0037] The goal of the generator is to generate data that is as realistic as possible to deceive the discriminator; the goal of the discriminator is to correctly distinguish between real data and generated data; the overall goal of generating adversarial distillation is to achieve Nash equilibrium through adversarial training between the generator and the discriminator, where the generator tries to minimize this function and the discriminator tries to maximize this function;

[0038]

[0039] Among them, the audio-visual global feature V that guides the network t ′,A t ′ is the real data sample, Pdata(V t ′) represents the distribution of real data; the visual features and auditory features V of the input generator s , A sAs noise, p(V s ), p(A s ) represents the distribution of noise; Generative adversarial distillation forces the features generated by the backbone network to approach the distribution of real features, and learns how to generate global features similar to the guided network from incomplete audio-visual observations;

[0040] Logits distillation; the logits output by the audiovisual guidance network are expressed as The logits output by the backbone network are expressed as Logits distillation loss combines the losses of soft labels and hard labels, uses L1 loss to calculate the difference in logits between the guide network and the backbone network, and uses MSE loss to calculate the difference between the backbone network prediction and the true label, which is specifically expressed as:

[0041]

[0042]

[0043] L LD =λ s T 2 L soft +λ h L hard

[0044] Where T is the distillation temperature used to smooth the logits distribution of the bootstrap network and the backbone network, Ns is the number of samples, and F label is the groundtruth provided by the dataset, ||g||1 represents the L1 loss operator, represents the MSE loss operator; λ is the weight coefficient, T 2 is to balance the scaling effect introduced by temperature; by minimizing the comprehensive loss function L LD , the backbone network can not only learn the logits information of the guidance network, but also maintain high-precision prediction between the real labels.

[0045] The method for calculating the total loss function in S6 to optimize the model parameters is:

[0046] L Total =μ1L FA +μ2L vGD +μ3L aGD +μ4L LD

[0047] Among them, μ is the weight factor used to balance the percentage of each part in the total loss function, L Total is the total loss function, L FA is the feature alignment loss, L vGD To generate the adversarial loss, LaGD is the logits distillation loss, L LD Estimate the loss for the location.

[0048] Compared with the prior art, the present invention has the following beneficial effects:

[0049] The present invention improves the global perception and missing information completion capabilities of the tracker through interactive learning between the backbone network and the guide network. A global feature reconstruction module based on a generative adversarial network is proposed. The generator globally reconstructs the features with local information missing in the backbone network. At the same time, a heterogeneous signal multi-level feature fusion attention module is also proposed. This module uses attention weights to dynamically fuse the local features of heterogeneous signals with the global reconstruction features, thereby effectively utilizing the complementarity and consistency of the multi-level features of heterogeneous signals. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0051] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0052] Figure 1 It is the overall framework diagram of the present invention;

[0053] Figure 2 This is a diagram of the distributed audio-visual sensor setup in the AV163 dataset;

[0054] Figure 3 The figure shows the comparison results between the single-modal signal method and the most advanced audio and video method on the standard dataset and the dataset with local information missing;

[0055] Figure 4 The figure shows the comparison results of ablation experiments on a standard dataset and a dataset with local modality loss;

[0056] Figure 5 The figure shows the comparison results of different loss combinations on the standard dataset and the dataset with local modal loss. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. These descriptions are only to further illustrate the features and advantages of the present invention, rather than to limit the claims of the present invention. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0058] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0059] An audio and video target tracking method based on global-local information interaction mechanism, such as Figure 1-5 As shown, the following steps are included:

[0060] S1. Obtain audio and video signals, input the global video signal into the visual guidance network, and input the full-channel audio signal into the auditory guidance network to extract global visual and auditory features at different levels.

[0061] Visually guided network: The purpose of the object tracking task is to select an arbitrary object for localization in the first frame of the video, so it is impossible to collect specific data in advance to train the detector for tracking. To this end, the visually guided network introduces a general deep metric learning method to train the model in the initial offline stage, converting the tracking problem into a similarity measure between the known object and the search area. The visual network adopts a Siamese network structure and calculates the similarity response map between the target template features and the search area features through cross-correlation operations.

[0062]

[0063] Where I t is the picture frame currently input to the network, I tem is the reference template, i.e. the user-defined tracking target in the first frame. is a convolutional neural network-based visual encoder with shared weights. * indicates that the convolution operation is used for similarity measurement. Response map V t It reflects the probability of the target appearing at each position in the search image and is used to provide visual positioning clues.

[0064] Auditory guidance network: The auditory guidance network consists of an acoustic localization clue extraction module and a codec-based deep feature extraction module, and its input is a multi-channel audio signal from a microphone array. First, the spatiotemporal acoustic localization clues are extracted, and the audiovisual space mapping is constructed based on the camera model to obtain a sampling grid of potential sound source positions at different depths in three-dimensional space. Then the GCF spectrum at the sampling grid is calculated through the coherence between audio signals. The depth at which the GCF mapping peaks is selected to generate a spatial GCF map sGCF. Considering the intermittent nature of the speech signal and the continuity of the speaker's movement, the sGCF map with the largest peak is selected over a period of time to generate a spatiotemporal GCF map stGCF. Subsequently, the extracted acoustic cues learn higher-dimensional and deep deep features in the codec-based network. Audio Feature A t ' is defined as follows:

[0065]

[0066] Among them, U t is the 16-channel audio signal from the microphone array, R Ω (g) represents the extraction of acoustic localization cues, Denotes the encoder and decoder of the auditory guided network, A t This hybrid feature extraction method uses both artificially designed prior knowledge and the powerful expression ability of neural networks, which improves feature quality and positioning effect.

[0067] S2. Input the local video signal and some channel audio signals into the backbone network to extract visual and auditory features with local information missing at different levels. The backbone network consists of a lightweight audio-visual encoder, a feature reconstruction module, a heterogeneous signal multi-level feature fusion attention module, and a prediction head. S2.1. The local features extracted by the audio-visual encoder are completed by the feature reconstruction module to complete the features with local information missing. S2.2. The audio-visual local features and the reconstructed global features are input into the fusion attention module for fusion. S2.3. The target positioning result is output through the prediction head.

[0068] S3. Construct a feature reconstruction module based on a generative adversarial network in the backbone network to reconstruct global features of features with local information missing. Generative adversarial networks (GANs) perform well in generating high-quality image and audio content. Inspired by the idea of ​​generative adversarial networks, we designed a feature reconstruction module in the backbone network, which is deployed on the visual and auditory branches respectively. First, a feature reconstruction module is designed in the backbone network, which is deployed on the visual and auditory branches respectively. First, the generator G is used to receive missing features and reconstruct global features. Then, the discriminator D is used to distinguish whether these features are reconstructed features or real global features from the guided network. The visual and auditory generators consist of an encoder-decoder, and the discriminator consists of multiple convolutional layers and fully connected layers. Let V s , A s are the visual and auditory features of the input generator, V′, A t ' is the global audio-visual feature to guide the network, and the discriminator is processed as follows:

[0069]

[0070] in: Represents the generated visual reconstruction features and auditory reconstruction features. Discriminator D v and D a The output of represents the probability that the input feature is a real feature. By alternately optimizing the generator and the discriminator, the generator gradually learns to reconstruct the missing features and assists the backbone network in processing incomplete observation data.

[0071] S4. Introduce a heterogeneous signal multi-level feature fusion attention module to achieve complementary and consistent utilization of heterogeneous signal multi-level features by integrating features from heterogeneous signals and different levels.

[0072] In order to make the tracker focus on the detailed features of the target in the audio-visual observation stream while not ignoring the background and context information, we propose a heterogeneous signal multi-level feature fusion attention module. First, the local features and the global reconstruction features are fused at multiple levels based on attention to obtain the global-local features. make The perceptual weights generated for the global-local attention module are higher in reliable observations and lower in ambiguous observations with visual occlusion and limited hearing. Then, position encoding is added to the global-local fusion features to improve the tracker's ability to capture and utilize spatial structure information by combining spatial position information. Finally, the feature fusion of audiovisual heterogeneous signals is performed through the heterogeneous signal cross-attention mechanism. The calculation process of the multi-level feature fusion attention module is as follows:

[0073]

[0074] Among them, MMCA stands for the heterogeneous signal cross attention mechanism, which consists of two multi-head attention modules, each of which uses one modal signal as the qurey and the other modal signal as the key and value. By calculating the correlation between heterogeneous signals, the information of different heterogeneous signals can be effectively interacted and integrated. av is the fused audio-visual feature. Pos represents the position code. The proposed heterogeneous signal multi-level feature fusion attention module balances the local and global information of audio-visual heterogeneous signals through two-step fusion.

[0075] S5. Through the global to local knowledge distillation architecture, the knowledge of the guide network is transferred to the backbone network, so that the backbone network can learn global features and improve the robustness to local information loss. The fused heterogeneous signal features are input into the prediction head for target positioning and generate target tracking results.

[0076] In order to effectively pass the global information processed by the guide network to the backbone network and make up for the shortcomings of the backbone network in processing incomplete signal information, we propose a global-local distillation method, which includes feature alignment distillation, generative adversarial distillation and logits distillation.

[0077] Feature alignment distillation: The output of the cross-correlation operation of the backbone network visual branch is represented as V s , specifically Among them I s It is the image frame with partial missing input by the backbone network. represents the visual encoder of the backbone network. The output of the backbone network auditory encoder is represented as A s , specifically Among them U s is the local channel audio signal from the microphone array, R Ω′ (g) represents the acoustic localization clues of the backbone network. Represents the auditory encoder of the backbone network. The feature alignment loss is based on the MSE loss and is divided into two parts: visual and auditory, expressed as:

[0078]

[0079] Where N is the dimension of the feature vector, Denotes the MSE loss operator. Feature alignment distillation improves the consistency of feature representation in the guided backbone network, ensuring that the backbone network can capture the main patterns and information of heterogeneous signals even if the input signal is incomplete.

[0080] Generative adversarial distillation: Generative adversarial distillation is based on the idea of ​​generative adversarial networks (GANs). Through adversarial training, the reconstructed features generated by the backbone network are as similar as possible to the real features extracted by the guide network, thereby improving the backbone network's ability to capture global information. Generative adversarial loss includes two parts: visual and auditory. The overall objective function is expressed as:

[0081] The goal of the generator is to generate data that is as realistic as possible to deceive the discriminator. The goal of the discriminator is to correctly distinguish between real data and generated data. The overall goal of generative adversarial distillation is to achieve a Nash equilibrium through adversarial training between the generator and the discriminator, where the generator tries to minimize this function and the discriminator tries to maximize this function.

[0082]

[0083] Among them, the audio-visual global feature V that guides the network t ′,A t ′ is the real data sample, Pdata(V t ′) represents the distribution of real data. The visual features and auditory features V s , A s As noise, p(V s ), p(A s ) represents the distribution of noise. Generative adversarial distillation forces the features generated by the backbone network to approach the distribution of real features, and learns how to generate global features similar to the guided network from incomplete audio-visual observations.

[0084] Logits distillation. The logits output by the audiovisual guidance network are expressed as The logits output by the backbone network are expressed as Logits distillation loss combines the losses of soft labels and hard labels, uses L1 loss to calculate the difference in logits between the guide network and the backbone network, and uses MSE loss to calculate the difference between the backbone network prediction and the true label, which is specifically expressed as:

[0085]

[0086] L LD =λ s T 2 L soft +λ h L hard

[0087] Where T is the distillation temperature used to smooth the logits distribution of the bootstrap network and the backbone network, Ns is the number of samples, and F label is the groundtruth provided by the dataset, ||g||1 represents the L1 loss operator, represents the MSE loss operator. λ is the weight coefficient, T 2 This is to balance the scaling effect introduced by temperature. By minimizing the comprehensive loss function L LD , the backbone network can not only learn the logits information of the guidance network, but also maintain high-precision prediction between the real labels.

[0088] S6. Optimize the model parameters by calculating the total loss function. The total loss includes feature alignment loss, generative adversarial loss, logits distillation loss, and position estimation loss.

[0089] L Total =μ1L FA +μ2L vGD +μ3L aGD +μ4L LD

[0090] Among them, μ is the weight factor used to balance the percentage of each part in the total loss function, L Total is the total loss function, L FA is the feature alignment loss, L vGD To generate the adversarial loss, L aGD is the logits distillation loss, L LD Estimate the loss for the location.

[0091] The present invention proposes an audio and video target tracking method and system based on a global-local information interaction mechanism, which is used for audio and video target tracking tasks with modality missing. The proposed tracking framework is driven by a guided distillation model, wherein the input of the audio-visual guidance model is global information, and the input of the backbone model has local information missing. By transferring the knowledge of the guidance network to the backbone model, the backbone model can better adapt to complex dynamic scenes with incomplete observation data. In the backbone network, a global feature reconstruction module based on a generative adversarial network is constructed to reconstruct global features of features with local information missing. Furthermore, multi-level feature fusion attention of heterogeneous signals is introduced, and the complementarity and consistency of heterogeneous signals and features at different levels are effectively utilized by integrating incomplete features and reconstructed features in the audio-visual network. Experimental results show that, Figure 3-5 As shown in the figure, Global refers to the standard dataset, Local refers to the dataset with local modality loss, the unit of MAE is pixel, AO refers to pure audio, VO refers to pure vision, and √ indicates that such loss is included in the calculation of the total loss. The proposed method outperforms the existing trackers in tracking on the standard dataset and the dataset with local information loss, proving its superiority and robustness under complex conditions.

[0092] Only the preferred embodiments of the present invention are described in detail above, but the present invention is not limited to the above embodiments. Various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention, and various changes should be included in the protection scope of the present invention.

Claims

1. An audio and video target tracking method based on a global-local information interaction mechanism, characterized in that: The following steps are involved: S1, obtain audio and video signals, input the global video signal into the visual guidance network, and input the full-channel audio signal into the auditory guidance network to extract global visual and auditory features at different levels; S2, inputting the local video signal and some channel audio signals into the backbone network to extract the visual and auditory features with local information missing at different levels; S3. Construct a feature reconstruction module based on a generative adversarial network in the backbone network to reconstruct global features of features with local information missing; S4, introduces a heterogeneous signal multi-level feature fusion attention module, which integrates features from heterogeneous signals and different levels to achieve complementary and consistent utilization of heterogeneous signal multi-level features; S5. Through the global to local knowledge distillation architecture, the knowledge of the guide network is transferred to the backbone network, so that the backbone network can learn global features and improve the robustness to local information loss. The fused heterogeneous signal features are input into the prediction head for target positioning and generate target tracking results. S6. Optimize the model parameters by calculating the total loss function. The total loss includes feature alignment loss, generative adversarial loss, logits distillation loss, and position estimation loss.

2. According to claim 1, the audio and video target tracking method based on the global-local information interaction mechanism is characterized in that: The method of guiding the network in S1 to extract visual and auditory features at different levels is as follows: visual guidance network: training the model in the initial offline stage to convert the tracking problem into a similarity measure between a known target and a search area; The visual network adopts the Siamese network structure and calculates the similarity response map between the target template features and the search area features through cross-correlation operations; Where I t is the picture frame currently input to the network, I tem is the reference template, i.e. the user-defined tracking target in the first frame; is a convolutional neural network-based visual encoder with shared weights; * indicates that the convolution operation is used for similarity measurement; the response map V t It reflects the probability of the target appearing at each position in the search image and is used to provide visual positioning clues; Auditory Guidance Network: The auditory guidance network consists of an acoustic localization cue extraction module and a codec-based deep feature extraction module, and its input is a multi-channel audio signal from a microphone array. First, it extracts acoustic localization cues based on time and space, builds an audio-visual space mapping based on the camera model, and obtains a sampling grid of potential sound source positions at different depths in three-dimensional space. Then, the GCF map at the sampling grid is calculated through the coherence between audio signals. The depth at which the GCF mapping peaks is selected to generate a spatial GCF mapping sGCF. Considering the intermittent nature of the speech signal and the continuity of the speaker's movement, the sGCF mapping with the largest peak over a period of time is selected to generate a spatiotemporal GCF mapping stGCF. Subsequently, the extracted acoustic cues learn higher-dimensional and deep-level deep features in the codec-based network. Audio FeaturesA t ' is defined as follows: Among them, U t is the 16-channel audio signal from the microphone array, R Ω (g) represents the extraction of acoustic localization cues, Denotes the encoder and decoder of the auditory guided network, A t Represents the result of the auditory guided encoder; this hybrid feature extraction method uses both manually designed prior knowledge and the powerful expression ability of neural networks to improve feature quality and positioning effect.

3. The method for tracking an audio and video target based on a global-local information interaction mechanism according to claim 1, characterized in that: The backbone network in S2 consists of a lightweight audio-visual encoder, a feature reconstruction module, a heterogeneous signal multi-level feature fusion attention module, and a prediction head.

4. The method for tracking an audio and video target based on a global-local information interaction mechanism according to claim 3, characterized in that: The method for extracting visual and auditory features with local information missing at different levels by the backbone network in S2 is: S2.1, the local features extracted by the audio-visual encoder are used to complete the features with missing local information through the feature reconstruction module; S2.2, the audio-visual local features and the reconstructed global features are input into the fusion attention module for fusion; S2.

3. Output the target positioning result through the prediction head.

5. The method for tracking audio and video targets based on global-local information interaction mechanism according to claim 1, characterized in that: The feature reconstruction method of the feature reconstruction module of the generative adversarial network in S3 is as follows: a feature reconstruction module is designed in the backbone network and deployed on the visual and auditory branches respectively; first, the generator G is used to receive the missing features and reconstruct the global features, and then the discriminator D is used to distinguish whether these features are reconstructed features or real global features from the guided network; The visual and auditory generators consist of encoders and decoders, and the discriminator consists of multiple convolutional layers and fully connected layers; let V s , A s are the visual and auditory features of the input generator, V′, A t ' is the global audio-visual feature to guide the network, and the discriminator is processed as follows: Where: V s g =G v (V s ), Represents the generated visual reconstruction features and auditory reconstruction features; discriminator D v and D a The output of represents the probability that the input feature is a true feature; by alternately optimizing the generator and the discriminator, the generator gradually learns to reconstruct the missing features and assists the backbone network in processing incomplete observation data.

6. The method for tracking audio and video targets based on a global-local information interaction mechanism according to claim 1, characterized in that: The method for multi-level feature fusion of heterogeneous signals in S4 is: First, the local features and the global reconstruction features are fused at multiple levels based on attention to obtain the global-local feature V s gl , make The perceptual weights generated for the global-local attention module are higher in reliable observations and lower in ambiguous observations with visual occlusion and limited hearing. Then, position encoding is added to the global-local fusion features to improve the tracker's ability to capture and utilize spatial structure information by combining spatial position information. Finally, the feature fusion of audiovisual heterogeneous signals is performed through the heterogeneous signal cross-attention mechanism. The calculation process of the multi-level feature fusion attention module is as follows: Among them, MMCA stands for the heterogeneous signal cross attention mechanism, which consists of two multi-head attention modules, one modal signal as qurey and the other modal signal as key and value. By calculating the correlation between heterogeneous signals, the information of different heterogeneous signals can be effectively interacted and integrated; F av is the fused audio-visual feature; Pos represents the position code; the proposed heterogeneous signal multi-level feature fusion attention module balances the local and global information of audio-visual heterogeneous signals through two-step fusion.

7. The method for tracking audio and video targets based on global-local information interaction mechanism according to claim 1, characterized in that: The method of the global to local knowledge distillation strategy in S5 is: the global to local knowledge distillation strategy includes feature alignment distillation, generative adversarial distillation and logits distillation; Feature alignment distillation: The output of the cross-correlation operation of the backbone network visual branch is represented as V s , specifically Among them I s It is the image frame with partial missing input by the backbone network. represents the visual encoder of the backbone network; the output of the backbone network auditory encoder is represented as A s , specifically Among them U s is the local channel audio signal from the microphone array, R Ω′ (g) represents the acoustic localization clues of the backbone network. represents the auditory encoder of the backbone network; the feature alignment loss is based on the MSE loss and is divided into two parts, visual and auditory, expressed as: Where N is the dimension of the feature vector, represents the MSE loss operator; feature alignment distillation improves the consistency of feature representation in the guided backbone network, ensuring that the backbone network can capture the main patterns and information of heterogeneous signals even if the input signal is incomplete; Generative adversarial distillation: Generative adversarial distillation is based on the idea of ​​generative adversarial networks (GANs). Through adversarial training, the reconstructed features generated by the backbone network are as similar as possible to the real features extracted by the guide network, thereby improving the backbone network's ability to capture global information. Generative adversarial loss includes two parts: visual and auditory. The overall objective function is expressed as: The goal of the generator is to generate data that is as realistic as possible to deceive the discriminator; the goal of the discriminator is to correctly distinguish between real data and generated data; the overall goal of generating adversarial distillation is to achieve Nash equilibrium through adversarial training between the generator and the discriminator, where the generator tries to minimize this function and the discriminator tries to maximize this function; Among them, the audio-visual global feature V that guides the network t ′,A t ′ is the real data sample, Pdata(V t ′) represents the distribution of real data; the visual features and auditory features V of the input generator s , A s As noise, p(V s ), p(A s ) represents the distribution of noise; Generative adversarial distillation forces the features generated by the backbone network to approach the distribution of real features, and learns how to generate global features similar to the guided network from incomplete audio-visual observations; Logits distillation; the logits output by the audiovisual guidance network are expressed as The logits output by the backbone network are expressed as Logits distillation loss combines the losses of soft labels and hard labels, uses L1 loss to calculate the difference in logits between the guide network and the backbone network, and uses MSE loss to calculate the difference between the backbone network prediction and the true label, which is specifically expressed as: L LD =λ s T 2 L soft +λ h L hard Where T is the distillation temperature used to smooth the logits distribution of the bootstrap network and the backbone network, Ns is the number of samples, and F label is the groundtruth provided by the dataset, ||g||1 represents the L1 loss operator, represents the MSE loss operator; λ is the weight coefficient, T 2 is to balance the scaling effect introduced by temperature; by minimizing the comprehensive loss function L LD , the backbone network can not only learn the logits information of the guidance network, but also maintain high-precision prediction between the real labels.

8. The method for tracking audio and video targets based on global-local information interaction mechanism according to claim 1, characterized in that: The method for calculating the total loss function in S6 to optimize the model parameters is: L Total =μ1L FA +μ2L vGD +μ3L aGD +μ4L LD Among them, μ is the weight factor used to balance the percentage of each part in the total loss function, L Total is the total loss function, L FA is the feature alignment loss, L vGD To generate the adversarial loss, L aGD is the logits distillation loss, L LD Estimate the loss for the location.

Images