Audio-visual speaker tracking method based on multi-stage multi-modal knowledge distillation strategy

Abstract

The invention belongs to the technical field of audio-visual speaker tracking, and particularly relates to an audio-visual speaker tracking method based on a multi-stage multi-modal knowledge distillation strategy, which comprises the following steps: S1, acquiring multi-modal data including visual data and auditory data, and performing data preprocessing on the multi-modal data; s2, inputting a visual teacher network to carry out visual feature extraction and speaker position estimation; s3, auditory feature extraction and speaker position estimation are carried out; s4, calculating feature collaborative loss in a feature extraction stage; s5, calculating fusion guide loss; s6, the Logs loss is calculated; and S7, calculating the total loss, and dynamically adjusting each loss weight according to the current learning state of the student network. According to the method, the audio-visual fusion process is guided by constructing the audio-visual teacher-student model and introducing the knowledge distillation normal form, the knowledge from the teacher network is gradually migrated to the student network, and the positioning capability and the fusion capability of the student network are improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of audio-visual speaker tracking, and in particular relates to an audio-visual speaker tracking method based on a multi-stage multimodal knowledge distillation strategy. Background Art

[0002] The goal of multimodal speaker tracking is to use multimodal data from sensors such as microphones and cameras to continuously determine the location of the speaker target at each time step. It has a wide range of applications in human-computer interaction, scene understanding, and intelligent security monitoring. In recent years, the advancement of audio-visual sensor fusion technology has promoted the research of speaker tracking based on audio-visual fusion, and many excellent audio-visual trackers have emerged. However, how to effectively integrate audio and visual data in a complementary way in complex scenes and how to improve tracking efficiency while maintaining tracking accuracy remain major challenges.

[0003] Existing audio-visual target localization and tracking algorithms can be divided into two categories: parameter-based methods and learning-based methods. A popular strategy of parameterized methods is the Bayesian tracker, which performs speaker localization by recursively estimating the probability distribution of the target state. In multimodal scenarios, the Bayesian tracker can use the joint distribution of audio and visual information to predict the position of the speaker and update it at each time step. The parameterized method has clear physical meaning and good interpretability, but it has poor robustness in noisy environments and complex scenes, and has high computational complexity.

[0004] Compared with parameter-based methods, learning-based methods use neural networks to learn the relationship between audio and video features and speaker positions from a large amount of data, and have higher robustness and adaptability in complex environments. However, existing methods implement multimodal information fusion in Bayesian filters and attention mechanisms, but lack attention to how to guide audio and video to complement each other effectively. In addition, the calculation process of multimodal fusion and particle filters is relatively complex, which affects the model tracking efficiency to a certain extent. Summary of the invention

[0005] In response to the above technical problems, the present invention provides a method of introducing audio-visual knowledge distillation paradigm to promote audio-visual information fusion in multiple stages and gradually transfer the knowledge from the teacher network to the student network, thereby improving the positioning and fusion capabilities of the student network. This is to solve the problem of how to effectively integrate audio and visual data in a complementary way in complex scenes, and how to improve tracking efficiency while maintaining tracking accuracy.

[0006] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0007] The audio-visual speaker tracking method based on a multi-stage multimodal knowledge distillation strategy includes the following steps:

[0008] S1, acquiring multimodal data, including image information from a camera as visual data and a multi-channel audio signal from a microphone array as auditory data, and then preprocessing the acquired multimodal data;

[0009] S2, input the preprocessed visual data into the visual teacher network for visual feature extraction and speaker position estimation;

[0010] S3, inputting the preprocessed auditory data into the auditory teacher network for auditory feature extraction and speaker position estimation;

[0011] S4, calculating feature synergy loss in the feature extraction stage;

[0012] S5, in the feature fusion stage, the visual features and auditory features learned and optimized in the feature extraction stage are sent to the fusion module, and then the fusion guidance loss is calculated;

[0013] S6. In the position estimation stage, the fused features are input into the prediction head for prediction, the final target tracking result is generated, and the Logits loss is calculated for effective knowledge transfer and output alignment;

[0014] S7. Calculate the total loss and dynamically adjust the weights of each loss according to the current learning status of the student network.

[0015] The preprocessing of multimodal data in S1 requires preprocessing of visual and auditory data respectively to obtain image samples and audio samples. The method for obtaining image samples is:

[0016] S11, given a video sequence number and a camera number, determine the start frame and the end frame of the image sample, and extract a template image;

[0017] S12, read the bounding box information from the groundtruth file and convert it into an index for training;

[0018] S13, traverse the video frames, obtain each frame image and its groundtruth information, and randomly crop to generate a search image;

[0019] S14. Store the template image, search image and groundtruth information in the dictionary imgData.

[0020] The method for obtaining the audio sample is:

[0021] S15, initializing parameters for processing audio data, creating a dictionary audioData containing audio data, and updating the corresponding audio data into the dictionary;

[0022] S16. Convert the image frame into an audio frame by loading the audio data and the image data, then calculate the GCC-PHAT of each audio frame and store the result in the dictionary GCCData.

[0023] The method of visual feature extraction and speaker position estimation in S2 is:

[0024] First, the target is selected in the starting frame of the image sample for positioning. Then all preprocessed image samples are passed through an AlexNet model encoder as a visual teacher encoder to initially extract the features of each image sample:

[0025] F vt =f t Venc (X t )

[0026] Among them, X t is the image frame input to the current visual teacher network, is the teacher visual encoder, F vt The visual features initially extracted by the teacher visual encoder are then passed to a siamese network to convert the target tracking task into a measure of the similarity between the known target and the area to be searched. The Siamese network contains two sub-networks with shared weights, one for processing the target template and the other for processing the area to be searched. The similarity response map between the features of the two sub-networks is calculated:

[0027] F v ' t =cor(X tem ,F vt ) = f t Vext (X tem )*f t Vext (F vt )

[0028] Among them, X tem is the template image, f Vext (g) is the teacher visual extractor used to extract deeper visual features. cor(g) represents the cross-correlation operation used to calculate similarity. This process enables the visual features to have positioning capabilities. Finally, the response map is projected into two dimensions through multiple linear layers to obtain the two-dimensional coordinates of the speaker’s mouth.

[0029] The method for extracting auditory features and estimating speaker position in S3 is:

[0030] The input of the auditory teacher network is a multi-channel audio signal collected from a microphone array. The network first estimates the existence probability of the sound source by analyzing the coherence of the sound signal at different time and spatial positions, thereby extracting acoustic localization information; first, it is assumed that is the sampling point on the image plane, and the generalized cross power phase transform (GCC-PHAT) is used to calculate the coherence between the audio signals received by multiple microphones, thereby generating a GCFmap denoted as GCF(Q 2d ,t); then project the model through the camera The sampling points on the image plane are converted into different depths D = {d1,...,d k ,…,d L}Sampling point GCFmaps of different depths are calculated, and the depth k of the peak sound source position is selected from these GCFmaps. m , the generated space GCFmap is recorded as Due to the discontinuity of the sound source signal, the first m2 largest spatial GCFmaps are selected within a certain time window [t-m1] to generate a spatiotemporal GCFmap denoted as Then, a pre-trained AlexNet encoder is used as the teacher auditory encoder to extract preliminary auditory features from the GCFmap.

[0031] F at =f t Aenc (U Ω (S t ))

[0032] Where S t represents the multi-channel audio signal input to the auditory teacher network, and U Ω (g) Extraction of acoustic positioning information, f t Aenc (g) is the teacher auditory encoder, F at The auditory features initially extracted by the teacher's auditory encoder are then passed through a linear auditory feature extraction module to further extract the possible sound source locations in the GCF graph.

[0033] F a ' t =f Aext (F at )

[0034] where f Vext (g) is the teacher auditory extractor, which is used to extract deeper auditory features so that the features have positioning capabilities. Finally, the features are projected into two dimensions through multiple linear layers to obtain the two-dimensional coordinates of the speaker’s mouth.

[0035] The process of the feature extraction stage in S4 is as follows:

[0036] S41, calculating the difference between the visual features initially generated by the visual teacher network and the features generated by the visual student network, and optimizing the ability of the visual student network to extract features;

[0037] S42, calculating the difference between the auditory features initially generated by the auditory teacher network and the features generated by the auditory student network, and optimizing the ability of the auditory student network to extract features;

[0038] The calculation formula for the student network to initially generate visual features and auditory features is as follows:

[0039] F vs =f s Venc (X s )

[0040] Where X s The image frame input to the current visual student network is f s Venc (g) is the student visual encoder, F vs The visual features initially extracted by the student visual encoder;

[0041] F as =f s Aenc (U Ω (S s ))

[0042] Where S s represents the multi-channel audio signal input to the auditory student network, and U Ω (g) Extraction of acoustic positioning information, f s Aenc (g) is the student auditory encoder, F as The auditory features initially extracted by the student auditory encoder;

[0043] The calculation formula of feature synergy loss is as follows:

[0044] L1=λ v MSE(F vt ||F vs )+λ a MSE(F at ||F as )

[0045] where MSE(g) represents the MSE loss operator, λ v ,λ a It represents the weight coefficient, which is used to adjust the utilization of visual and auditory information at different stages.

[0046] The process of the feature fusion stage in S5 is as follows:

[0047] S51, the difference between the deeper visual features and the audio-visual fusion features of the computational vision teacher network;

[0048] S52, calculate the difference between the deeper auditory features of the auditory teacher network and the audio-visual fusion features;

[0049] The calculation formula of the fusion guidance loss is as follows:

[0050] L2=λ v MSE(F v ' t ||F av )+λ a MSE(F a ' t ||F av )

[0051] where F av represents the features after the audio-visual fusion module, MSE(g) represents the MSE loss operator, and λ v ,λ a It represents the weight coefficient, which is used to adjust the utilization of visual and auditory information at different stages.

[0052] The process of the position estimation stage in S6 is as follows:

[0053] The calculation formula for Logits distillation loss is as follows:

[0054]

[0055] where MSE(g) represents the MSE loss operator, represents the Logits used by the visual teacher network to predict the speaker's mouth coordinates, represents the Logits used by the auditory teacher network to predict the speaker's mouth coordinates, represents the Logits of the audiovisual student network used to predict the speaker's mouth coordinates, λ v ,λ a It represents the weight coefficient, which is used to adjust the utilization of visual and auditory information at different stages.

[0056] The process of calculating the total loss in S7 is:

[0057] The total loss of the model is divided into four parts: feature synergy loss, fusion guidance loss, Logits distillation loss, and the Euclidean distance between the speaker's mouth coordinates predicted by the student network and the actual speaker's mouth coordinates. The calculation formula for the total loss is as follows:

[0058]

[0059] Among them, C label represents the speaker's mouth position coordinates provided in the AV16.3 dataset, C pred represents the coordinates of the mouth position of the speaker predicted by the student network, and D represents the number of samples. α and β represent weight factors used to balance the contribution of each loss in the total loss function.

[0060] Compared with the prior art, the present invention has the following beneficial effects:

[0061] The present invention proposes to guide the audio-visual fusion process by constructing an audio-visual teacher-student model and introducing a knowledge distillation paradigm. In addition, the present invention also proposes a multi-stage progressive knowledge distillation framework, including three different distillation losses. In the feature extraction stage, the feature alignment loss ensures that the feature representation of the student network is consistent with that of the teacher network output by aligning the features of the teacher-student network that has passed through the feature encoder. In the feature fusion stage, the fusion guidance loss guides the multimodal fusion process of the student network by aligning the enhanced features of the teacher with the audio-visual fusion features of the student network. In the position estimation stage, the Logits distillation loss helps the student model better capture the speaker positioning features by aligning the positioning features of the teacher-student. Through multi-stage progressive knowledge distillation, the knowledge from the teacher network is gradually transferred to the student network, and the capabilities of the student network are gradually enhanced, ensuring that the tracking efficiency is improved without sacrificing the tracking accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the implementation methods of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the implementation methods or the description of the prior art. Obviously, the drawings in the following description are only exemplary, and for ordinary technicians in this field, other implementation drawings can be derived from the provided drawings without creative work.

[0063] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0064] Figure 1 It is the overall framework diagram of the present invention;

[0065] Figure 2 The figure shows the comparison results between the single-modal method and the most advanced audio-visual method on the AV163 dataset;

[0066] Figure 3 This is a comparison result diagram of different loss combinations of the present invention on the AV163 dataset;

[0067] Figure 4 A comparison chart of the tracking results of the present invention on the AV163 dataset. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. These descriptions are only to further illustrate the features and advantages of the present invention, rather than to limit the claims of the present invention. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0069] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0070] like Figure 1-4 As shown, an audio-visual speaker tracking method based on a multi-stage multimodal knowledge distillation strategy includes the following steps:

[0071] Step S1: Acquire multimodal data, including image information from a camera as visual data and a multi-channel audio signal from a microphone array as auditory data, and then pre-process the acquired multimodal data.

[0072] Step S2: Input the preprocessed visual data into the visual teacher network to extract visual features at different levels and obtain the two-dimensional coordinates of the speaker's mouth.

[0073] Step S3: Input the preprocessed auditory data into the auditory teacher network to extract auditory features at different levels and obtain the two-dimensional coordinates of the speaker's mouth.

[0074] Step S4: Feature synergy loss is used in the feature extraction stage to supervise the student network in the early stage of feature extraction to ensure that the feature representation of the student network is consistent with that of the teacher network after passing through the feature encoder.

[0075] Step S5: In the feature fusion stage, the visual features and auditory features learned and optimized in the feature extraction stage are sent to the fusion module, the fusion of the two modal features is supervised by the fusion guidance loss, and the enhanced features of the teacher network are used to guide the multimodal fusion process in the student network to maximize the complementary advantages of audio-visual fusion, thereby optimizing the output of the fusion module.

[0076] Step S6: In the position estimation stage, the fused features are input into the prediction head for prediction to generate the final target tracking result. The effective knowledge transfer and output alignment of Logits loss are used to help the student model better capture the target positioning features.

[0077] Step S7: Compare the output of the student network with the target tracking result, and dynamically adjust the weight of the distillation loss according to the current learning state of the student network. By monitoring the learning progress of the student network, the distillation loss weights of the three stages of feature extraction, feature fusion and position estimation are adaptively allocated at different training stages to better guide the learning of the student network. Ensure that the multimodal fusion system can maintain good tracking performance even when there is a loss or noise in a single visual or auditory modality.

[0078] Further, in step S1, preprocessing the multimodal data requires preprocessing the visual and auditory data respectively to obtain image samples and audio samples. The step of obtaining the image samples specifically includes:

[0079] S11 Let the video sequence be S = {X1, X2, ..., X N}, where X t Represents the image of the tth frame. Given the sequence number and camera number, determine the starting frame fv of the image sample start and the end frame fv end , template image X tem Get X from the start frame image tem =X fstart ; S12 reads the bounding box information B from the groundtruth file of the corresponding dataset t , index the groundtruth information into B t ′, so as to be used in the subsequent training process; S13 traverses the video frames to obtain each frame image and groundtruth information, and randomly crops each frame image to generate a search image; S14 converts the template image X tem , search image X t , groundtruth information B t ' is stored in the dictionary 'imgData':

[0080] imgData = {'template':X tem,'search':X t ,'ground':B t '};

[0081] The steps of obtaining audio samples specifically include: S15 initializing the parameters required for processing audio data, creating a dictionary 'audioData' containing audio data, where the key is a combination of the sequence name and the camera number, and the value is the corresponding audio data: audioData = {'seq, cam': DATA}, updating the audio data to 'audioData'; S16 calculating GCC-PHAT, loading the audio data from the saved 'audioData', and obtaining the starting frame fa of each sequence audio sample start and end frame fa end As well as the camera audio data, the audio frame needs to be converted through the image frame calculation: fa = 2 × fv-2-m3, m3 = 15 to ensure the consistency of image samples and audio samples. The GCC-PHAT of each audio frame fa is calculated and stored in the dictionary 'GCCData'.

[0082] Further, in step S2, the visual teacher network includes a visual encoder and a siamese network, and the steps of extracting visual features at different levels and obtaining the two-dimensional coordinates of the speaker's mouth specifically include:

[0083] First, the target is selected in the starting frame of the image sample for positioning. Then all preprocessed image samples are passed through an AlexNet model encoder as a visual teacher encoder to initially extract the features of each image sample:

[0084] F vt =f t Venc (X t ) (1)

[0085] In formula (1), X t The image frame input to the current teacher network is f t Venc (g) is the teacher visual encoder, F vt The visual features initially extracted by the visual encoder are then passed to a siamese network to convert the target tracking task into a measure of the similarity between the known target and the area to be searched. The Siamese network contains two sub-networks with shared weights, one for processing the target template and the other for processing the area to be searched. The similarity response map between the features of the two sub-networks is calculated:

[0086] F v ' t =cor(X tem,F vt ) = f t Vext (X tem )*f t Vext (F vt ) (2)

[0087] In formula (2), X tem is the template image, f Vext (g) is the teacher visual extractor used to extract deeper visual features. cor(g) represents the cross-correlation operation used to calculate similarity. This process enables the visual features to have positioning capabilities. Finally, the response map is projected into two dimensions through multiple linear layers to obtain the two-dimensional coordinates of the speaker’s mouth.

[0088] Further, in step S3, the auditory teacher network includes an acoustic localization clue extraction module, an auditory encoder and an auditory extractor, and the steps of extracting visual features at different levels and obtaining the two-dimensional coordinates of the speaker's mouth specifically include:

[0089] The input of the auditory teacher network is a multi-channel audio signal collected from a microphone array. The network first estimates the existence probability of the sound source by analyzing the coherence of the sound signal at different time and spatial positions, thereby extracting acoustic localization information. The specific process is as follows: First, assume is the sampling point on the image plane, and the generalized cross power phase transform (GCC-PHAT) is used to calculate the coherence between the audio signals received by multiple microphones, thereby generating a GCFmap denoted as GCF(Q 2d ,t). Then the model is projected through the camera The sampling points on the image plane are converted into different depths D = {d1, ..., d k ,…,d L}Sampling point GCFmaps of different depths are calculated, and the depth k of the peak sound source position is selected from these GCFmaps. m , the generated space GCFmap is recorded as Due to the discontinuity of the sound source signal, the first m2 largest spatial GCFmaps are selected within a certain time window [t-m1] to generate a spatiotemporal GCFmap denoted as Then, a pre-trained AlexNet encoder is used as the teacher auditory encoder to extract preliminary auditory features from the GCFmap.

[0090] F at =f t Aenc (U Ω (S t )) (3)

[0091] In formula (3), S t represents the multi-channel audio signal input to the auditory teacher network, and U Ω (g) Extraction of acoustic positioning information, f t Aenc (g) is the teacher auditory encoder, F at The auditory features initially extracted by the teacher's auditory encoder are then passed through a linear auditory feature extraction module to further extract the possible sound source locations in the GCF graph.

[0092] F a ' t =f Aext (F at ) (4)

[0093] In formula (4), f Vext (g) is the teacher auditory extractor, which is used to extract deeper auditory features so that the features have positioning capabilities. Finally, the features are projected into two dimensions through multiple linear layers to obtain the two-dimensional coordinates of the speaker’s mouth.

[0094] Furthermore, in step S4, the process of calculating the feature synergy loss specifically includes:

[0095] Before the multimodal fusion mechanism, it is hoped that the features initially extracted by the visual and auditory student network can be regularized to ensure that these features can be effectively aligned and consistent with the features of the teacher network. To this end, a feature distillation loss is proposed. S41 calculates the difference between the visual features initially generated by the visual teacher network and the features generated by the visual student network, and optimizes the ability of the visual student network to extract features. S42 calculates the difference between the auditory features initially generated by the auditory teacher network and the features generated by the auditory student network, and optimizes the ability of the auditory student network to extract features.

[0096] The student network gradually approaches the performance of the teacher network when extracting visual and auditory features, ensuring that subsequent multimodal fusion can be performed on higher quality initial features. The calculation formula for the student network to initially generate visual and auditory features is as follows:

[0097] F vs =f s Venc (X s ) (5)

[0098] In formula (5), X s The image frame input to the current visual student network is f s Venc (g) is the student visual encoder, F vs The visual features initially extracted by the student visual encoder;

[0099] F as =f s Aenc (U Ω (S s )) (6)

[0100] In formula (6), S s represents the multi-channel audio signal input to the auditory student network, and U Ω (g) Extraction of acoustic positioning information, f s Aenc (g) is the student auditory encoder, F as The auditory features initially extracted by the student auditory encoder;

[0101] The characteristic distillation loss is calculated as follows:

[0102] L1=λ v MSE(F vt ||F vs )+λ a MSE(F at ||F as ) (7)

[0103] In formula (7), MSE(g) represents the MSE loss operator, λ v ,λ a It represents the weight coefficient, which is used to adjust the utilization of visual and auditory information at different stages.

[0104] Furthermore, in step S5, the feature fusion process specifically includes:

[0105] In order to make full use of the positioning information contained in the deep features of the teacher network, it is hoped that in the multimodal fusion process, the fused features of the student network can be effectively aligned with the deep features of the teacher network, thereby guiding the collaborative attention mechanism to better perform the complementarity between the modalities. For this purpose, the fusion guidance loss is designed. By calculating the alignment loss between the fused features of the student network and the deep features of the teacher network, the multimodal fusion process is further optimized to ensure that the fused features can effectively capture the complementarity of audio and visual data, thereby achieving higher robustness and accuracy in the speaker tracking task. The calculation formula of the fusion guidance loss is as follows:

[0106] L2=λ v MSE(F v ' t ||F av )+λ a MSE(F a ' t ||F av ) (8)

[0107] In formula (8), F av represents the features after the audio-visual fusion module, MSE(g) represents the MSE loss operator, and λ v ,λ a It represents the weight coefficient, which is used to adjust the utilization of visual and auditory information at different stages.

[0108] Further, in step S6, the process of the position estimation stage specifically includes:

[0109] In order to enhance the prediction ability of the student network, we hope that the output of the student network in the Logits layer is closer to the performance of the teacher network, helping the student network to better capture the tracking details of the teacher network during the learning process. For this purpose, the Logits distillation loss is designed. By calculating the Logits alignment loss between the student network and the teacher network, the output layer of the student network is fine-tuned, and the student network is prompted to gradually calibrate the prediction distribution of the teacher network so that the student network can more comprehensively inherit the knowledge and tracking ability of the teacher network. The calculation formula of the Logits distillation loss is as follows:

[0110]

[0111] In formula (9), MSE(g) represents the MSE loss operator, represents the Logits used by the visual teacher network to predict the speaker's mouth coordinates, represents the Logits used by the auditory teacher network to predict the speaker's mouth coordinates, represents the Logits of the audiovisual student network used to predict the speaker's mouth coordinates, λ v ,λ a It represents the weight coefficient, which is used to adjust the utilization of visual and auditory information at different stages.

[0112] Further, in step S7, the process of calculating the total loss specifically includes:

[0113] The total loss of the model is divided into four parts: feature distillation loss, fusion guidance loss, Logits distillation loss, and the Euclidean distance between the speaker's mouth coordinates predicted by the student network and the actual speaker's mouth coordinates. The calculation formula for the total loss is as follows:

[0114]

[0115] In formula (10), C label represents the speaker's mouth position coordinates provided in the AV16.3 dataset, C pred represents the coordinates of the mouth position of the speaker predicted by the student network, and D represents the number of samples. α and β represent weight factors used to balance the contribution of each loss in the total loss function.

[0116] The present invention proposes to guide the audio-visual fusion process by constructing an audio-visual teacher-student model and introducing a knowledge distillation paradigm. In addition, the present invention also proposes a multi-stage progressive knowledge distillation framework, including three different distillation losses. In the feature extraction stage, the feature alignment loss ensures that the feature representation of the student network is consistent with that of the teacher network output by aligning the features of the teacher-student network that has passed through the feature encoder. In the feature fusion stage, the fusion guidance loss guides the multimodal fusion process of the student network by aligning the enhanced features of the teacher with the audio-visual fusion features of the student network. In the position estimation stage, the Logits distillation loss helps the student model better capture the speaker positioning features by aligning the positioning features of the teacher-student. Through multi-stage progressive knowledge distillation, the knowledge from the teacher network is gradually transferred to the student network, and the capabilities of the student network are gradually enhanced, ensuring that the tracking efficiency is improved without sacrificing the tracking accuracy.

[0117] Only the preferred embodiments of the present invention are described in detail above, but the present invention is not limited to the above embodiments. Various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention, and various changes should be included in the protection scope of the present invention.

Claims

1. An audio-visual speaker tracking method based on a multi-stage multimodal knowledge distillation strategy, characterized in that: The following steps are involved: S1, acquiring multimodal data, including image information from a camera as visual data and a multi-channel audio signal from a microphone array as auditory data, and then preprocessing the acquired multimodal data; S2, input the preprocessed visual data into the visual teacher network for visual feature extraction and speaker position estimation; S3, inputting the preprocessed auditory data into the auditory teacher network for auditory feature extraction and speaker position estimation; S4, calculating feature synergy loss in the feature extraction stage; S5, in the feature fusion stage, the visual features and auditory features learned and optimized in the feature extraction stage are sent to the fusion module, and then the fusion guidance loss is calculated; S6. In the position estimation stage, the fused features are input into the prediction head for prediction, the final target tracking result is generated, and the Logits loss is calculated for effective knowledge transfer and output alignment; S7. Calculate the total loss and dynamically adjust the weights of each loss according to the current learning status of the student network.

2. The audiovisual speaker tracking method based on multi-stage multimodal knowledge distillation strategy according to claim 1 is characterized in that: The preprocessing of multimodal data in S1 requires preprocessing of visual and auditory data respectively to obtain image samples and audio samples. The method for obtaining image samples is: S11, given a video sequence number and a camera number, determine the start frame and the end frame of the image sample, and extract a template image; S12, read the bounding box information from the groundtruth file and convert it into an index for training; S13, traverse the video frames, obtain each frame image and its groundtruth information, and randomly crop to generate a search image; S14. Store the template image, search image and groundtruth information in the dictionary imgData.

3. The audiovisual speaker tracking method based on multi-stage multimodal knowledge distillation strategy according to claim 2 is characterized in that: The method for obtaining the audio sample is: S15, initializing parameters for processing audio data, creating a dictionary audioData containing audio data, and updating the corresponding audio data into the dictionary; S16. Convert the image frame into an audio frame by loading the audio data and the image data, then calculate the GCC-PHAT of each audio frame and store the result in the dictionary GCCData.

4. The audiovisual speaker tracking method based on multi-stage multimodal knowledge distillation strategy according to claim 1, characterized in that: The method of visual feature extraction and speaker position estimation in S2 is: First, select the target in the starting frame of the image sample for positioning. Then, all preprocessed image samples are passed through an AlexNet model encoder as a visual teacher encoder to initially extract the features of each image sample: F vt =f t Venc (X t ) Among them, X t The image frame input to the current visual teacher network is f t Venc (g) is the teacher visual encoder, F vt The visual features initially extracted by the teacher visual encoder are then passed to a siamese network to convert the target tracking task into a measure of the similarity between the known target and the area to be searched. The Siamese network contains two sub-networks with shared weights, one for processing the target template and the other for processing the area to be searched. The similarity response map between the features of the two sub-networks is calculated: F v ′ t =cor(X tem ,F vt )=f t Vext (X tem )*f t Vext (F vt ) Among them, X tem is the template image, f Vext (g) is the teacher visual extractor used to extract deeper visual features. cor(g) represents the cross-correlation operation used to calculate similarity. This process enables the visual features to have positioning capabilities. Finally, the response map is projected into two dimensions through multiple linear layers to obtain the two-dimensional coordinates of the speaker’s mouth.

5. The audiovisual speaker tracking method based on multi-stage multimodal knowledge distillation strategy according to claim 1, characterized in that: The method for extracting auditory features and estimating speaker position in S3 is: The input of the auditory teacher network is a multi-channel audio signal collected from a microphone array. The network first estimates the existence probability of the sound source by analyzing the coherence of the sound signal at different time and spatial positions, thereby extracting acoustic localization information; first, it is assumed that is the sampling point on the image plane, and the generalized cross power phase transform (GCC-PHAT) is used to calculate the coherence between the audio signals received by multiple microphones, thereby generating a GCFmap denoted as GCF(Q 2d ,t); then project the model through the camera The sampling points on the image plane are converted into different depths D = {d1,...,d k ,…,d L }Sampling point GCFmaps of different depths are calculated, and the depth k of the peak sound source position is selected from these GCFmaps. m , the generated space GCFmap is recorded as Due to the discontinuity of the sound source signal, the first m2 largest spatial GCFmaps are selected within a certain time window [t-m1] to generate a spatiotemporal GCFmap denoted as Then, a pre-trained AlexNet encoder is used as the teacher auditory encoder to extract preliminary auditory features from the GCFmap. F at =f t Aenc (U Ω (S t )) Where S t represents the multi-channel audio signal input to the auditory teacher network, and U Ω (g) Extraction of acoustic positioning information, f t Aenc (g) is the teacher auditory encoder, F at The auditory features initially extracted by the teacher's auditory encoder are then passed through a linear auditory feature extraction module to further extract the possible sound source locations in the GCF graph. F a ′ t =f Aext (F at ) where f Vext (g) is the teacher auditory extractor, which is used to extract deeper auditory features so that the features have positioning capabilities. Finally, the features are projected into two dimensions through multiple linear layers to obtain the two-dimensional coordinates of the speaker’s mouth.

6. The audiovisual speaker tracking method based on multi-stage multimodal knowledge distillation strategy according to claim 1, characterized in that: The process of the feature extraction stage in S4 is as follows: S41, calculating the difference between the visual features initially generated by the visual teacher network and the features generated by the visual student network, and optimizing the ability of the visual student network to extract features; S42, calculating the difference between the auditory features initially generated by the auditory teacher network and the features generated by the auditory student network, and optimizing the ability of the auditory student network to extract features; The calculation formula for the student network to initially generate visual features and auditory features is as follows: F vs =f s Venc (X s ) Where X s The image frame input to the current visual student network is f s Venc (g) is the student visual encoder, F vs The visual features initially extracted by the student visual encoder; F as =f s Aenc (U Ω (S s )) Where S s represents the multi-channel audio signal input to the auditory student network, and U Ω (g) Extraction of acoustic positioning information, f s Aenc (g) is the student auditory encoder, F as The auditory features initially extracted by the student auditory encoder; The calculation formula of feature synergy loss is as follows: L1=λ v MSE(F vt ||F vs )+λ a MSE(F at ||F as ) where MSE(g) represents the MSE loss operator, λ v ,λ a It represents the weight coefficient, which is used to adjust the utilization of visual and auditory information at different stages.

7. The audiovisual speaker tracking method based on multi-stage multimodal knowledge distillation strategy according to claim 1, characterized in that: The process of the feature fusion stage in S5 is as follows: S51, the difference between the deeper visual features and the audio-visual fusion features of the computational vision teacher network; S52, calculate the difference between the deeper auditory features of the auditory teacher network and the audio-visual fusion features; The calculation formula of the fusion guidance loss is as follows: L2=λ v MSE(F v ′ t ||F av )+λ a MSE(F a ′ t ||F av ) Among them, F av represents the features after the audio-visual fusion module, MSE(g) represents the MSE loss operator, and λ v ,λ a It represents the weight coefficient, which is used to adjust the utilization of visual and auditory information at different stages.

8. The audiovisual speaker tracking method based on multi-stage multimodal knowledge distillation strategy according to claim 1, characterized in that: The process of the position estimation stage in S6 is as follows: The calculation formula for Logits distillation loss is as follows: Where MSE(g) represents the MSE loss operator, represents the Logits used by the visual teacher network to predict the speaker's mouth coordinates, represents the Logits used by the auditory teacher network to predict the speaker's mouth coordinates, represents the Logits of the audiovisual student network used to predict the speaker's mouth coordinates, λ v ,λ a It represents the weight coefficient, which is used to adjust the utilization of visual and auditory information at different stages.

9. The audiovisual speaker tracking method based on multi-stage multimodal knowledge distillation strategy according to claim 1, characterized in that: The process of calculating the total loss in S7 is: The total loss of the model is divided into four parts: feature synergy loss, fusion guidance loss, Logits distillation loss, and the Euclidean distance between the speaker's mouth coordinates predicted by the student network and the actual speaker's mouth coordinates. The calculation formula for the total loss is as follows: Among them, C label represents the speaker's mouth position coordinates provided in the AV16.3 dataset, C pred represents the coordinates of the mouth position of the speaker predicted by the student network, and D represents the number of samples. α and β represent weight factors used to balance the contribution of each loss in the total loss function.

Images