A speech and face cross-modal association method based on mutual information

Abstract

The application relates to the technical field of speech and face cross-modal correlation, and discloses a speech and face cross-modal correlation method based on mutual information, which comprises the following steps: a basic experiment framework is constructed by using a pretreated VoxCeleb1 data set; a category balance sampling strategy is adopted to load the VoxCeleb1 data set; intermediate features of a speech mode and intermediate features of a face mode are output; cross-modal common semantic factors are extracted; a gating feature selection unit is designed; fine-grained feature filtering is carried out; a cross-modal feature alignment network based on hash learning is adopted; a generator-discriminator adversarial learning mechanism is constructed to constrain the modal invariance of the features; a cross-modal identity correlation loss function is proposed to constrain the identity correlation degree of the hash code; and a small batch training strategy is adopted for training. Compared with existing work, the accuracy of the method in speech-face cross-modal matching, verification and retrieval tasks is significantly improved.

Description

Technical Field

[0001] This invention relates to the field of cross-modal association technology between speech and face, specifically a method for cross-modal association between speech and face based on mutual information. Background Technology

[0002] In the current era of big data, the fusion and collaborative understanding of multimodal data (text, images, audio, and video) has become a research hotspot. Speech and face, as the most biometric and emotionally expressive modal pair, demonstrate unique value in scenarios such as intelligent security, human-computer interaction, and cross-media retrieval. Speech signals contain rich temporal dynamic information, such as identity features, semantic content, and intonation changes, while faces provide visual information such as identity, micro-expressions, and lip movement features. The complementarity of the two can enhance the robust perception capabilities of the system in complex scenarios.

[0003] However, existing cross-modal association methods face serious challenges. There is a heterogeneous gap between the spectral features of speech and the spatial representation of faces. Traditional single-modal models and shallow contrastive learning are difficult to model global semantic consistency across modalities. Furthermore, noise interference in real-world scenarios (such as background noise and face occlusion) further weakens the reliability of cross-modal matching.

[0004] Meanwhile, existing cross-modal speech-face association learning methods mainly adopt a supervised learning paradigm, relying on labeled datasets to guide the network in feature learning. This typically involves two stages: first, using a pre-trained feature extraction network to acquire deep features for both speech and face modalities; then, feature alignment is performed through one or two fully connected layers; and finally, metric learning loss functions (such as CenterLoss, TripletLoss, N-pairLoss, LiftedStructureLoss, etc.) are used to optimize intra-class feature aggregation and inter-class feature separation. Most researchers tend to improve the loss function, but often overlook the crucial role of feature extraction in subsequent feature alignment. While pre-trained features contain single-modal discriminative information, they still carry modality-specific noise. If shallow nonlinear projection is directly used to force cross-modal spatial matching, the model struggles to extract highly discriminative cross-modal identity representations from heterogeneous features. Summary of the Invention

[0005] (a) Technical problems to be solved To address the shortcomings of existing technologies, this invention provides a cross-modal association method for speech and faces based on mutual information. This method can accurately retrieve face images corresponding to specific identities from an image database, given only a segment of unknown speech. By constructing an adversarial training network architecture and combining it with a strategy to maximize inter-modal mutual information, latent semantic alignment is achieved under unsupervised conditions. This algorithm effectively utilizes unlabeled data to generate pseudo-supervisory signals to guide network training, achieving cross-modal 1:N matching, verification, and retrieval tasks. Experiments have verified that our model significantly improves accuracy in these three tasks compared to existing methods.

[0006] (II) Technical Solution To achieve the above objectives, the present invention provides the following technical solution: a method for cross-modal association of speech and face based on mutual information, comprising the following steps: Step 1: Use the preprocessed VoxCeleb1 dataset to build a basic experimental framework, providing a data foundation for cross-modal association; Step 2: Load the VoxCeleb1 dataset using a class-balanced sampling strategy; Step 3: Output intermediate features of the speech modality ; Step 4: Output intermediate features of the face modality ; Step 5: From speech features With facial features Extracting cross-modal common semantic factors and ; Step 6: Design a gating feature selection unit Fine-grained feature filtering is then performed. Step 7: Cross-modal feature alignment network based on hash learning; Step 8: Construct a generator-discriminator adversarial learning mechanism to constrain the modality invariance of features; Step 9: Propose a cross-modal identity association loss function to constrain the identity association degree of the hash code; Step 10: Employ a mini-batch training strategy, selecting M data tuples in each iteration. During training, the feature extraction network inputs the features of the two modalities into their respective subnetworks to complete feature extraction. The resulting features are used by subsequent networks to learn a compact hash code representation, denoted as... The k-Means algorithm is used to cluster hash codes to generate pseudo-labels to guide network training.

[0007] Preferably, in step one, the VoxCeleb1 dataset includes video data from 1225 independent speakers, which can be represented as follows: ,in Indicates the first A video of the speaker. ; Regarding the first A video of a speaker, from which multiple audio segments and facial image frames are extracted, can be represented as follows: ,in, Representing the The first speaker's video A voice clip, The range is from 1 to That is, the first There are a total of [number] speakers. One voice sample; similarly, Representing the The first speaker's video Personal face image frames, The range is from 1 to That is, the first There are a total of [number] speakers. Personal face image frame; The VoxCeleb1 dataset is divided into a training set, a test set, and a validation set.

[0008] Preferably, in step two, a class-balanced sampling strategy is used to load the VoxCeleb1 dataset into training batches. Each training batch dynamically constructs a subset containing multiple different speaker identities, and randomly samples Q sets of cross-modal sample pairs for each identity. Each cross-modal sample pair contains a speech segment and its corresponding face image. Network training is performed on each training batch. During network training, the data of each iteration is presented in the form of tuples. ,in Indicates a speech segment, This represents a face image, and the M parameter controls the amount of data used in each iteration of training.

[0009] Preferably, in step three, the ECAPA-TDNN network pre-trained on the VoxCeleb2 dataset is used as the speech feature extraction network. For a speech segment, its MFCC features are first extracted and normalized, then used as input to the ECAPA-TDNN network to extract features, and the output is the intermediate features of the speech modality. ; Step four involves using the Inception-V1 network pre-trained on the VGGFace2 dataset as the face feature extraction network. Facial images are transmitted via the internet. It will output the intermediate features of the face modality. .

[0010] Preferably, in step five, the cross-modal common semantic factor and It is expressed as follows: ; ; in, , This is the weight matrix. , It is the bias vector; Design a mutual information estimation network It is used to supervise the training process of cross-modal common feature selection networks; The network uses concatenated vectors of common features between speech and face. As input features, they are represented as follows: ; in and These represent the outputs of the speech and facial common feature selection network, respectively. Operators enable horizontal concatenation of features; It is a three-layer fully connected neural network that outputs a scalar score, represented as follows: ; ; ; in, It is a non-linear activation function, defined by the right-hand side: ; After establishing the basic architecture of the mutual information estimation network, a dual distribution of the feature space is constructed based on the pseudo-identity labels, as follows: ; ; in, For cross-modal joint distribution feature pairs with the same identity, These are cross-modal marginal distribution feature pairs of different identities; The lower bound estimate of mutual information is constructed through variational inference, as shown in the following formula: ; Constructing the mutual information loss function , means as follows: ; Preferably, in step six, the gating feature selection unit An adaptive weight generator is used to weight the feature dimensions based on their importance; its mathematical expression is as follows: ; in, This represents the Sigmoid activation function. It is a learnable gated weight matrix. This is the bias vector.

[0011] Preferably, in step seven, the cross-modal feature alignment network... The features of the two modalities obtained through the gated feature selection unit are used as inputs to the network, which automatically feeds them into the corresponding feature alignment subnetwork. The network structure uses multiple fully connected layers. To achieve feature alignment and reduce internal covariate bias, a batch normalization layer is immediately following each fully connected layer, except for the last fully connected layer. Batch normalization can be expressed as follows: ; in, This represents the features after passing through the fully connected layer. and These represent the mean and variance of all samples within a batch in each feature channel, respectively. It is a small constant used for numerical stability to prevent division by zero. and These are learnable parameters used to scale and translate normalized features to restore the model's expressive power and allow the network to learn the optimal representation of the normalized features. Simultaneously, residual connections are made between the normalized features of the first layer and the features of the last layer, fusing features from early and deep layers. This allows for effective integration of features captured by the early layers and high-level features learned by the deep layers. The two modalities in the early layers are trained independently, while the later layers share weights for both modalities, enabling the cross-modal network to learn more general feature representations. This network maps features to a 128-dimensional space to obtain embedded features. ; ; in, This represents the speech feature alignment subnetwork and the face feature alignment subnetwork in the constructed cross-modal feature alignment network. These represent the parameters learned by the two sub-networks in the cross-modal feature alignment network, respectively. On the Internet Add a hash function layer at the last level. Output a fixed-length hash code For each sample or The hash code obtained through the network can be represented as: ; Where d represents the length of the hash code; The final output of the network can be represented as: ; ; For mini-batch training, the hash code features obtained in each iteration can be represented as: ; in, This refers to the hash code representation of speech or face obtained by a hash-based cross-modal feature alignment network in each iteration. The matrix represents the number of speech or face modality samples in each iteration. Each row vector in the dataset, whether it's a speech sample or a face sample, has a length of [length missing]. The hash code is a hash code, and each dimension of the hash code takes a value of +1 or -1.

[0012] Preferably, step eight, designing the discriminator network, is... As a "modal origin discriminator", its input is the continuous embedded features output by the cross-modal feature alignment network. and , Output single-dimensional score: ; The training objective is to maximize the discrimination score of face embedding and minimize the discrimination score of speech embedding. The optimization objective of the discriminator can be expressed as: ; A gradient penalty mechanism based on Lipschitz continuity constraints is introduced to enhance the training stability of the discriminator. The mathematical expression is as follows: ; in, It is a penalty coefficient used to balance the strength of constraints. From and Randomly sampled samples It is the L2 norm. It is a discriminator pair The gradient of the discriminator loss function is expressed as follows: ; Simultaneously, align speech features with subnetworks Defined as a generator, whose optimization objective is adversarial to that of the discriminator, the generator adversarial loss is expressed as follows: ; Preferably, in step nine, the cross-modal identity association loss function uses the NWJ variational lower bound to estimate cross-modal mutual information. The theoretical basis for this is: for two random variables... and Mutual information satisfies the inequality: ; The identity association of hash codes is constrained by maximizing the lower bound of cross-modal mutual information, and the loss function is defined as: ; The first term drives the aggregation of identical identity hash code pairs in the embedding space, while the second term suppresses spurious associations between different identity hash code pairs. The similarity between voice and face hash code pairs is represented as follows: ; in, The length of the hash code. For the temperature hyperparameter, the total loss function is expressed as follows: ; in, This is a hyperparameter.

[0013] Preferably, step ten is implemented as follows: During the training iteration, the hash codes of isomorphic tuples are concatenated along their feature dimensions to construct aggregated samples. The number of samples required for clustering is reduced from 2M in the case of bimodal independent processing to M. At the beginning of each training round, clustering is performed based on the aggregated samples, and the clustering results are used as pseudo-labels. Then, the loss function is calculated and backpropagation is performed to update the network parameters. By minimizing the loss function, a more discriminative hash code is generated. In the next training round, feature concatenation is re-performed based on the updated hash code to generate new aggregated samples. This is achieved through a continuous iterative process of "clustering to generate pseudo-labels → pseudo-labels to guide network optimization → updating hash codes".

[0014] Compared with existing technologies, this invention provides a method for cross-modal association of speech and face based on mutual information, which has the following beneficial effects: 1. This invention proposes a mutual information estimation method based on neural networks. By maximizing the mutual information between different modalities, it extracts common features between speech and face modalities, providing a better learning foundation for subsequent feature alignment networks, enabling them to better learn the intrinsic relationships between different modalities.

[0015] 2. This invention introduces a modality adaptive gating module to achieve dynamic feature selection and information filtering. It performs soft screening of input features dimension by dimension through learnable gating weights, suppressing task-irrelevant noise information while retaining cross-modal shared features.

[0016] 3. This invention proposes an adversarial training framework in which the generator guides the network to generate embedding features with modality invariance through adversarial loss, while the discriminator distinguishes between speech and face samples through modality classification loss. Through the adversarial optimization between the two, the distribution difference between speech and face features in the latent space is gradually reduced, which significantly alleviates the semantic gap problem caused by modality heterogeneity.

[0017] 4. This invention proposes a cross-modal alignment loss function based on the lower bound of NWJ mutual information, which enhances the discriminative ability of hash codes by maximizing the mutual information correlation between hash codes of the same identity while minimizing the implicit correlation of cross-identity samples. Attached Figure Description

[0018] Figure 1 This is the overall network framework diagram of the present invention; Figure 2 This is a diagram illustrating the steps of the method of the present invention; Figure 3 This is a visualization of the pre-learning feature distribution of the hash code using t-SNE in an embodiment of the present invention. Figure 4 This is a visualization of the feature distribution of hash codes after t-SNE learning in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Please see Figures 1-2 A method for cross-modal association of speech and face based on mutual information includes the following steps: Step 1: Construct a basic experimental framework using the preprocessed VoxCeleb1 dataset to provide a data foundation for cross-modal association. The VoxCeleb1 dataset includes video data from 1225 independent speakers, which can be represented as... ,in Indicates the first A video of the speaker. ; Regarding the first A video of a speaker, from which multiple audio segments and facial image frames are extracted, can be represented as follows: ,in, Representing the The first speaker's video A voice clip, The range is from 1 to That is, the first There are a total of [number] speakers. One voice sample; similarly, Representing the The first speaker's video Personal face image frames, The range is from 1 to That is, the first There are a total of [number] speakers. Personal face image frame; The VoxCeleb1 dataset was divided into a training set, a test set, and a validation set. Step 2: The VoxCeleb1 dataset is loaded into training batches using a class-balanced sampling strategy. Each training batch dynamically constructs subsets containing multiple different speaker identities, and Q sets of cross-modal sample pairs are randomly sampled for each identity. Each cross-modal sample pair contains a speech segment and its corresponding face image, effectively ensuring sample diversity during training. Network training is performed on each training batch. During network training, the data for each iteration is presented in tuple form. ,in Indicates a speech segment, The M parameter represents a face image, and controls the amount of data used in each iteration of training. Step 3: Use the ECAPA-TDNN network pre-trained on the VoxCeleb2 dataset as the speech feature extraction network. For a speech segment, its MFCC features are first extracted and normalized, then used as input to the ECAPA-TDNN network to extract features, and the output is the intermediate features of the speech modality. ; Step 4: Use the Inception-V1 network pre-trained on the VGGFace2 dataset as the face feature extraction network. Facial images are transmitted via the internet. It will output the intermediate features of the face modality. ; Step 5: From speech features With facial features Extracting cross-modal common semantic factors and A common feature selection network is designed, employing a linearly parameterized projection layer to avoid truncation of the feature distribution caused by nonlinear activation functions (such as ReLU), thus preventing the loss of low-frequency semantic information and cross-modal common semantic factors. and It is expressed as follows: ; ; in, , This is the weight matrix. , It is the bias vector: To effectively supervise the training process of the cross-modal common feature selection network, a mutual information estimation network with multi-level nonlinear mapping capability is designed. It models the difference between the joint distribution and marginal distribution of cross-modal common feature pairs through neural networks, forcing the common feature selection network to extract cross-modal common features; The network uses concatenated vectors of common features between speech and face. As input features, they are represented as follows: ; in and These represent the outputs of the speech and facial common feature selection network, respectively. Operators enable horizontal concatenation of features; It is a three-layer fully connected neural network that outputs a scalar score, represented as follows: ; ; ; in, It is a non-linear activation function, defined by the right-hand side: ; After establishing the basic architecture of the mutual information estimation network, a dual distribution of the feature space is constructed based on identity pseudo-labels to implement the contrastive learning strategy, as follows: ; ; in, For cross-modal joint distribution feature pairs with the same identity, These are cross-modal marginal distribution feature pairs of different identities; The lower bound estimate of mutual information is constructed through variational inference, as shown in the following formula: ; The goal is to maximize mutual information across modalities. In other words, it maximizes the joint distribution of the same identity and minimizes the marginal distribution between different identities. That is to say, it encourages the network to learn features that are shared by the network rather than features that are specific to the network. The training of deep learning models essentially involves optimizing model parameters by minimizing a loss function; therefore, the mutual information loss function... , means as follows: ; Step 6: To further filter information and select features, design a gated feature selection unit. This module performs fine-grained feature filtering by employing an adaptive weight generator to weight the feature dimensions based on their importance. Its mathematical expression is as follows: ; in, This represents the Sigmoid activation function. It is a learnable gated weight matrix. It is the bias vector; Selecting network output based on common features or As input, by introducing a residual connection structure, the integrity of key information is ensured while suppressing noise features, effectively solving the gradient vanishing problem caused by traditional hard thresholding filtering methods; Step 7: Cross-modal feature alignment network based on hash learning. The features of the two modalities obtained through the gated feature selection unit are used as inputs to the network, which automatically feeds them into the corresponding feature alignment subnetwork. The network structure uses multiple fully connected layers. To achieve feature alignment and reduce internal covariate bias, a batch normalization layer is immediately following each fully connected layer, except for the last fully connected layer. Batch normalization can be expressed as follows: ; in, This represents the features after passing through the fully connected layer. and These represent the mean and variance of all samples within a batch in each feature channel, respectively. It is a small constant used for numerical stability to prevent division by zero. and These are learnable parameters used to scale and translate normalized features to restore the model's expressive power and allow the network to learn the optimal representation of the normalized features. Simultaneously, residual connections are made between the normalized features of the first layer and the features of the last layer, fusing features from early and deep layers. This allows for effective integration of features captured by the early layers and high-level features learned by the deep layers. The two modalities in the early layers are trained independently, while the later layers share weights for both modalities, enabling the cross-modal network to learn more general feature representations. This network maps features to a 128-dimensional space to obtain embedded features. ; ; in, This represents the speech feature alignment subnetwork and the face feature alignment subnetwork in the constructed cross-modal feature alignment network. These represent the parameters learned by the two sub-networks in the cross-modal feature alignment network, respectively. On the Internet Add a hash function layer at the last level. Output a fixed-length hash code For each sample or The hash code obtained through the network can be represented as: ; Where d represents the length of the hash code; The final output of the network can be represented as: ; ; The learned hash codes enable voices and faces with the same identity to have a shorter Hamming distance in the hash space, while voices and faces with different identities have a longer distance. With mini-batch training, the hash code features obtained in each iteration can be represented as: ; in, This refers to the hash code representation of speech or face obtained by a hash-based cross-modal feature alignment network in each iteration. The matrix represents the number of speech or face modality samples in each iteration. Each row vector in the dataset, whether it's a speech sample or a face sample, has a length of [length missing]. The hash code, and each dimension of the hash code takes a value of +1 or -1; Step 8: To eliminate the "modal gap" between speech and facial modalities caused by differences in acquisition media and feature distribution, a generator-discriminator adversarial learning mechanism is constructed to constrain the modal invariance of features, and a discriminator network is designed. As a "modal origin discriminator", its input is the continuous embedded features output by the cross-modal feature alignment network. and , Output single-dimensional score: ; The training objective is to maximize the discrimination score of face embedding and minimize the discrimination score of speech embedding. The optimization objective of the discriminator can be expressed as: ; A gradient penalty mechanism based on Lipschitz continuity constraints is introduced to enhance the training stability of the discriminator. The mathematical expression is as follows: ; in, It is a penalty coefficient used to balance the strength of constraints. From and Randomly sampled samples It is the L2 norm. It is a discriminator pair The gradient of the discriminator is constrained by the L2 norm of the discriminator gradient to approach 1, thus forcing the discriminator to satisfy Lipschitz continuity. The discriminator loss function is expressed as follows: ; Simultaneously, align speech features with subnetworks Defined as a generator, whose optimization objective is adversarial to that of the discriminator, the generator adversarial loss is expressed as follows: ; By embedding the generator's output speech into the discriminator to achieve a high score, the generator is forced to learn a representation homogeneous with the facial modality feature distribution. Finally, through... and The alternating game optimization involves the discriminator learning to distinguish modes and the generator learning fuzzy modes, driving the cross-modal feature alignment network to learn deep features in the embedding space that have both identity discrimination and modality invariance. This provides a modality-invariant feature foundation for subsequent cross-modal matching, verification and retrieval based on hash learning. Step Nine: Cross-Modal Identity Association Loss Function. To achieve compact aggregation of voice-face features with the same identity in the hash space and inter-class discretization of features with different identities, we propose a cross-modal identity association loss function to constrain the identity association degree of the hash code. This loss uses the NWJ variational lower bound to estimate cross-modal mutual information, based on the theoretical basis that for two random variables... and Mutual information satisfies the inequality: ; The identity association of hash codes is constrained by maximizing the lower bound of cross-modal mutual information, and the loss function is defined as: ; The first term drives the aggregation of identical identity hash code pairs in the embedding space, while the second term suppresses spurious associations between different identity hash code pairs. The similarity between voice and face hash code pairs is represented as follows: ; in, The length of the hash code. For the temperature hyperparameter, the total loss function is expressed as follows: ; in, For hyperparameters; Step 10: Employ a mini-batch training strategy, selecting M data tuples in each iteration. During training, the feature extraction network inputs the features of the two modalities into their respective subnetworks to complete feature extraction. The resulting features are used by subsequent networks to learn a compact hash code representation, denoted as... To achieve unsupervised learning, the k-Means algorithm is used to cluster hash codes to generate pseudo-labels to guide network training. Specifically, during training iterations, the hash codes of isomorphic tuples are concatenated along their feature dimensions to construct aggregated samples. This method compresses the number of samples required for clustering from 2M in independent bimodal processing to M. This not only significantly reduces the sample size and computational complexity in the clustering stage but also enables the clustering process to effectively utilize complementary information from both modes, improving pseudo-label accuracy. At the beginning of each training round, clustering is performed based on the aggregated samples, and the clustering results are used as pseudo-labels. Subsequently, the loss function is calculated, and backpropagation is performed to update the network parameters. By minimizing the loss function, a more discriminative hash code is generated. In the next training round, feature concatenation is re-performed based on the updated hash code to generate new aggregated samples. Through continuous iterative processes of "clustering to generate pseudo-labels → pseudo-labels guiding network optimization → updating hash codes," the discriminative power of the hash codes is continuously enhanced, and the accuracy of the pseudo-labels is improved simultaneously.

[0021] Example 1 All experiments in this embodiment were performed on an NVIDIA GeForce RTX 4090 graphics card, using the PyTorch deep learning framework and Python 3.9 as the development foundation. During model training, the Adam optimizer was used with a learning rate of 1e-3. The parameter is set to 0.3. The value was set to 10 to optimize the model's learning ability and generalization performance. Finally, the Voxceleb dataset was used, split into training, validation, and test sets, for training, and the network's performance was tested on cross-modal matching, retrieval, and validation tasks.

[0022] Cross-modal 1:2 matching. Select one sample in the speech modality, and two samples in the face image modality. Find face images belonging to the same identity as the speech sample, and vice versa. Using the face images as the query benchmark, find speech segments that match their identities. We conducted this experiment on two different test sets: 'U' represents the test set not categorized by gender, and 'G' is the test set categorized by gender. By calculating the matching accuracy, we validated the model's performance. The comparative experiments are shown in the table below.

[0023] Table 1. Accuracy of cross-modal 1:2 matching task ① Cross-modal verification Given a voice sample and a face sample, determine whether they belong to the same person. This task uses the area under the curve (AUC) as the evaluation metric, and compares it with previous work as follows.

[0024] Table 2. AUC values ​​for cross-modal validation tasks ② Cross-modal retrieval Given a test sample of a certain modality, it is necessary to query positive samples that match the test sample from the total dataset. This task will use mean average precision (mAP) as the evaluation metric, and compare it with previous work as follows.

[0025] Table 3 Performance of mAP in cross-modal retrieval ③ Ablation test To quantitatively verify the gains of each module in cross-modal feature alignment accuracy and identity association performance, we conducted ablation experiments on cross-modal matching, verification, and retrieval tasks.

[0026] ④ Visualization of online learning like Figures 3-4 As shown, this paper further utilizes the t-SNE algorithm to perform 2D visualization of the learned high-dimensional face-voice hash code, where data points of the same color come from the same identity sample. Figure 3 The feature distribution before training is shown. It can be clearly seen that, due to the heterogeneity of the two modal data, the speech samples and face samples are roughly divided into two parts, and the distribution is relatively scattered. Figure 4 The results shown demonstrate the effectiveness of training with the proposed method. We not only eliminate intermodal heterogeneity, making two modal samples with the same identity closer to each other, but also keep samples with different identities further apart, indicating that the proposed method can effectively learn the correlation between modalities.

[0027] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for cross-modal association of speech and face based on mutual information, characterized in that, Includes the following steps: Step 1: Use the preprocessed VoxCeleb1 dataset to build a basic experimental framework, providing a data foundation for cross-modal association; Step 2: Load the VoxCeleb1 dataset using a class-balanced sampling strategy; Step 3: Output intermediate features of the speech modality ; Step 4: Output intermediate features of the face modality ; Step 5: From speech features With facial features Extracting cross-modal common semantic factors and ; Step 6: Design a gating feature selection unit Fine-grained feature filtering is then performed. Step 7: Cross-modal feature alignment network based on hash learning; Step 8: Construct a generator-discriminator adversarial learning mechanism to constrain the modality invariance of features; Step 9: Propose a cross-modal identity association loss function to constrain the identity association degree of hash codes. Construct the cross-modal identity association loss through the NWJ variational lower bound, obtain the sample pair similarity using the hash code similarity calculation formula, and obtain the total loss function by combining mutual information loss and adversarial loss to constrain the identity association degree of hash codes. The cross-modal identity association loss function uses the NWJ variational lower bound to estimate cross-modal mutual information. Its theoretical basis is that, for two random variables... and Mutual information satisfies the inequality: The identity association of hash codes is constrained by maximizing the lower bound of cross-modal mutual information, and the loss function is defined as follows: The first term drives the aggregation of identical identity hash code pairs in the embedding space, while the second term suppresses spurious associations between different identity hash code pairs. The similarity between voice and face hash code pairs is represented as follows: in, The length of the hash code. For the temperature hyperparameter, the total loss function is expressed as follows: in, For hyperparameters, The NWJ cross-modal identity association loss is constructed based on the variational lower bound of NWJ mutual information and calculated using hash code similarity. To combat the loss, a generator-discriminator adversarial learning mechanism is used, with a speech feature alignment sub-network serving as the generator and the discriminator distinguishing the source of the feature modality. The mutual information loss term is the negative estimate of the mutual information of cross-modal common semantic factors; The lower bound estimate of mutual information is constructed through variational inference, as shown in the following formula: Constructing the mutual information loss function , means as follows: Align speech features with subnetwork Defined as a generator, whose optimization objective is adversarial to that of the discriminator, the generator adversarial loss is expressed as follows: ； Step 10: Employ a mini-batch training strategy, selecting M data tuples in each iteration. During training, the feature extraction network inputs the features of the two modalities into their respective subnetworks to complete feature extraction. The resulting features are used by subsequent networks to learn a compact hash code representation, denoted as... The k-Means algorithm is used to cluster hash codes to generate pseudo-labels to guide network training.

2. The method for cross-modal association of speech and face based on mutual information according to claim 1, characterized in that: In step one, the VoxCeleb1 dataset includes video data from 1225 independent speakers, which can be represented as follows: ,in Indicates the first A video of the speaker. ; Regarding the first A video of a speaker, from which multiple audio segments and facial image frames are extracted, can be represented as follows: ,in, Representing the The first speaker's video A voice clip, The range is from 1 to That is, the first There are a total of [number] speakers. One voice sample; similarly, Representing the The first speaker's video Personal face image frames, The range is from 1 to That is, the first There are a total of [number] speakers. Personal face image frame; The VoxCeleb1 dataset is divided into a training set, a test set, and a validation set.

3. The method for cross-modal association of speech and face based on mutual information according to claim 2, characterized in that: In step two, a class-balanced sampling strategy is used to load the VoxCeleb1 dataset into training batches. Each training batch dynamically constructs a subset containing multiple different speaker identities, and randomly samples Q sets of cross-modal sample pairs for each identity. Each cross-modal sample pair contains a speech segment and its corresponding face image. Network training is performed on each training batch. During network training, the data for each iteration is presented in the form of tuples. ,in Indicates a speech segment, This represents a face image, and the M parameter controls the amount of data used in each iteration of training.

4. The method for cross-modal association of speech and face based on mutual information according to claim 3, characterized in that: In step three, the ECAPA-TDNN network pre-trained on the VoxCeleb2 dataset is used as the speech feature extraction network. For a speech segment, its MFCC features are first extracted and normalized, then used as input to the ECAPA-TDNN network to extract features, and the output is the intermediate features of the speech modality. ; Step four involves using the Inception-V1 network pre-trained on the VGGFace2 dataset as the face feature extraction network. Facial images are transmitted via the internet. It will output the intermediate features of the face modality. .

5. The method for cross-modal association of speech and face based on mutual information according to claim 4, characterized in that, In step five, cross-modal common semantic factors and It is expressed as follows: in, , This is the weight matrix. , It is the bias vector; Design a mutual information estimation network It is used to supervise the training process of cross-modal common feature selection networks; The network uses concatenated vectors of common features between speech and face. As input features, they are represented as follows: in and These represent the outputs of the speech and facial common feature selection network, respectively. Operators enable horizontal concatenation of features; It is a three-layer fully connected neural network that outputs a scalar score, represented as follows: in, It is a non-linear activation function, defined by the right-hand side: ; After establishing the basic architecture of the mutual information estimation network, a dual distribution of the feature space is constructed based on the pseudo-identity labels, as follows: in, For cross-modal joint distribution feature pairs with the same identity, These are cross-modal marginal distribution feature pairs with different identities.

6. The method for cross-modal association of speech and face based on mutual information according to claim 5, characterized in that, Step six, gated feature selection unit An adaptive weight generator is used to weight the feature dimensions based on their importance; its mathematical expression is as follows: in, This represents the Sigmoid activation function. It is a learnable gated weight matrix. This is the bias vector.

7. The method for cross-modal association of speech and face based on mutual information according to claim 6, characterized in that, Step seven involves a cross-modal feature alignment network. The features of the two modalities obtained through the gated feature selection unit are used as inputs to the network, which automatically feeds them into the corresponding feature alignment subnetwork. The network structure uses multiple fully connected layers. To achieve feature alignment and reduce internal covariate bias, a batch normalization layer is immediately following each fully connected layer, except for the last fully connected layer. Batch normalization can be expressed as follows: in, This represents the features after passing through the fully connected layer. and These represent the mean and variance of all samples within a batch in each feature channel, respectively. It is a small constant used for numerical stability to prevent division by zero. and These are learnable parameters used to scale and translate normalized features to restore the model's expressive power and allow the network to learn the optimal representation of the normalized features. Simultaneously, residual connections are made between the normalized features of the first layer and the features of the last layer, fusing features from early and deep layers. This allows for effective integration of features captured by the early layers and high-level features learned by the deep layers. The two modalities in the early layers are trained independently, while the later layers share weights for both modalities, enabling the cross-modal network to learn more general feature representations. This network maps features to a 128-dimensional space to obtain embedded features. in, This represents the speech feature alignment subnetwork and the face feature alignment subnetwork in the constructed cross-modal feature alignment network. These represent the parameters learned by the two sub-networks in the cross-modal feature alignment network, respectively. On the Internet Add a hash function layer at the last level. Output a fixed-length hash code For each sample or The hash code obtained through the network can be represented as: Where d represents the length of the hash code; The final output of the network can be represented as: For mini-batch training, the hash code features obtained in each iteration can be represented as: in, This refers to the hash code representation of speech or face obtained by a hash-based cross-modal feature alignment network in each iteration. The matrix represents the number of speech or face modality samples in each iteration. Each row vector in the dataset, whether it's a speech sample or a face sample, has a length of [length missing]. The hash code is a hash code, and each dimension of the hash code takes a value of +1 or -1.

8. The method for cross-modal association of speech and face based on mutual information according to claim 7, characterized in that, Step eight, designing the discriminator network. As a "modal origin discriminator", its input is the continuous embedded features output by the cross-modal feature alignment network. and , Output single-dimensional score: The training objective is to maximize the discrimination score of face embedding and minimize the discrimination score of speech embedding. The optimization objective of the discriminator can be expressed as: A gradient penalty mechanism based on Lipschitz continuity constraints is introduced to enhance the training stability of the discriminator. The mathematical expression is as follows: in, It is a penalty coefficient used to balance the strength of constraints. From and Randomly sampled samples It is the L2 norm. It is a discriminator pair The gradient of the discriminator loss function is expressed as follows: 。 9. The method for cross-modal association of speech and face based on mutual information according to claim 8, characterized in that, Step ten is implemented as follows: During the training iteration, the hash codes of isomorphic tuples are concatenated along their feature dimensions to construct aggregated samples. The number of samples required for clustering is reduced from 2M in the case of bimodal independent processing to M. At the beginning of each training round, clustering is performed based on the aggregated samples, and the clustering results are used as pseudo-labels. Then, the loss function is calculated and backpropagation is performed to update the network parameters. By minimizing the loss function, a more discriminative hash code is generated. In the next training round, feature concatenation is re-performed based on the updated hash code to generate new aggregated samples. This is achieved through a continuous iterative process of "clustering to generate pseudo-labels → pseudo-labels to guide network optimization → updating hash codes".

Images