A cross-modal speech and face association method based on a heterogeneous hash network

Abstract

The application belongs to the technical field of communication, and discloses a cross-modal speech and face association method based on a heterogeneous hash network, and the specific technical scheme is as follows: a cross-modal deep residual alignment network based on hash learning is proposed, the network maps input features into hash codes with fixed length, effectively reduces the feature dimension and the use of memory, meanwhile, the hashization greatly improves the calculation efficiency, the deep network integrated with the residual structure has stronger feature representation capability and good generalization, by minimizing the loss function, the speech and face features of the same identity can be made closer in the Hamming distance in the hash space, and the speech and face features of different identities are farther away, so that the feature alignment of the speech and the face is achieved, and the cross-modal matching, verification and retrieval tasks are completed, and the accuracy of the model in the cross-modal matching, retrieval and verification tasks is obviously improved compared with the existing method.

Description

Technical Field

[0001] This invention belongs to the field of communication technology, specifically relating to a cross-modal speech and face association method based on heterogeneous hash networks. Background Technology

[0002] In modern society, identity recognition is a core requirement in many fields such as security, criminal investigation, surveillance, and personal services. Traditional recognition methods, such as those based on passwords, ID cards, or biometrics (fingerprints, facial recognition, etc.), may meet the needs of identity verification to a certain extent, but they may have limitations in some situations, such as in scenarios where physical contact or other information is lacking.

[0003] Most existing cross-modal speech-face matching algorithms use feature extraction networks to extract features. In the feature alignment part, one or two fully connected layers map the features of the two modalities to the same feature space, minimizing the loss function to achieve intra-class compactness and inter-class expansion. However, using too few fully connected layers may lead to overfitting on the training data, resulting in insufficient generalization ability and inadequate feature alignment. Furthermore, during the mapping of original features to the same dimensional space through the network, each sample feature is represented as a high-dimensional floating-point number, which undoubtedly increases memory usage and computational burden when dealing with large-scale datasets. Summary of the Invention

[0004] To address the technical problems existing in the prior art, this invention proposes a cross-modal deep residual alignment network based on hash learning. The network maps input features to fixed-length hash codes, effectively reducing feature dimensionality and memory usage, while hashing significantly improves computational efficiency. The deep network integrating residual structures enhances feature representation capabilities and exhibits excellent generalization. By minimizing the loss function, the network can make speech and face features with the same identity closer in Hamming distance in the hash space, and speech and face features with different identities farther apart, thereby achieving feature alignment between speech and face and completing cross-modal matching, verification, and retrieval tasks.

[0005] To achieve the above objectives, the technical solution adopted by this invention is as follows: a cross-modal speech and face association method based on heterogeneous hash networks, the specific steps of which are as follows:

[0006] Dataset Construction: The network was trained and tested using the pre-processed VoxCeleb dataset. This voice-face dataset includes voice clips and face images for 1225 identities, represented as follows: Where i represents the i-th identity, and N = 1225. Each identity is internally represented as follows: in, Let j represent the j-th speech segment of the i-th identity, where j ranges from 1 to m, meaning the i-th identity has a total of m speech samples; similarly, The i-th identity has a j-th face image frame, where j ranges from 1 to n, meaning there are n face image frames for the i-th identity. To ensure a fair comparison, the frames are divided into a training set, a test set, and a validation set, following the partitioning method used in previous work.

[0007] Loading the dataset: Since triplet loss is needed for backpropagation to optimize network parameters, a sufficient number of triples (anchor, positive, negative) need to be selected in each iteration. Therefore, a sampler is added to ensure that the model receives diverse samples during training. Each iteration contains samples with multiple different identities, and each identity has a fixed number of instances, in order to select more effective triples for subsequent triplet loss calculation. During network training, the data is presented as a tuple in each iteration, specifically... Where v represents a speech segment, f represents a face image, and l is a label, if l i =l j , indicating v i ,f i With v j ,f j Belonging to the same person, if l i ≠l j These are people with different identities, and batch represents the number of tuples in each iteration of the network.

[0008] Construct a speech feature extraction network A v (·): The ECAPA-TDNN network, pre-trained on the VoxCeleb2 dataset, is used as the speech feature extraction network. For a speech segment, its MFCC features are first extracted and normalized, and then used as the input of the ECAPA-TDNN network to extract features. The output is a 192-dimensional vector as the final speech feature.

[0009] Construct a face feature extraction network A F (·): The Inception-V1 network, pre-trained on the VGGFace2 dataset, is used as the face feature extraction network. The face image is processed by the network and outputs a 512-dimensional vector as the final face feature.

[0010] A cross-modal feature alignment network φ(·,θ)={φv(·,θv),φf(·,θf)} is constructed: the features of the two modalities obtained in the feature extraction stage are used as inputs to this network, and the network automatically inputs the features of the two modalities into their corresponding feature alignment sub-networks. In terms of network structure design, multiple fully connected layers FC(·) are used to achieve feature alignment. To reduce internal covariate shift, except for the last fully connected layer, each fully connected layer is followed by a batch normalization layer BN(·). Batch normalization can be represented as follows:

[0011]

[0012] Where, x i μ represents the features after passing through the fully connected layer. B σ represents the mean of all sample features in a batch across each feature channel. B ε represents the variance of all sample features in each feature channel within a batch. ε is a small constant used for numerical stability to prevent division by zero. γ and β are learnable parameters used to scale and translate the normalized features to restore the model's expressive power and allow the network to learn the optimal representation of the normalized features. Simultaneously, the batch-normalized features of the first layer are residually connected to the features of the last layer, fusing early and deep features, allowing effective integration of features captured by the early layers with high-level features learned by the deep layers. The two modalities in the early layers are trained independently, while the later layers share weights for both modalities, enabling the cross-modal network to learn more general feature representations. This network maps the original 512-dimensional face features and 192-dimensional speech features to a 128-dimensional space. A hash function layer sgn(·) is added to the last layer of the network, causing the network to output a fixed-length hash code. Here, sgn(·) is a sign function that returns 1 if the element is positive and -1 otherwise. The learned hash code representation enables voices and faces with the same identity to have a shorter Hamming distance in the hash space, while voices and faces with different identities have a longer distance. The final output of the network is:

[0013] V = sgn(φ v (A v (v i ),θ v )

[0014] F = sgn(φ) f (A f (f i ),θ f )

[0015] Where, φ v φ represents the speech feature alignment subnetwork in the constructed cross-modal feature alignment network. fA represents the face feature alignment subnetwork in the constructed cross-modal feature alignment network. v (v i A represents the speech features obtained through a feature extraction network. f (f i θ represents the facial features obtained through a feature extraction network. v θ f F and V represent the parameters learned by the two sub-networks in the cross-modal feature alignment network, respectively, and F and V are the hash codes of the two modalities obtained in the end.

[0016] Constructing the loss function: Based on the proposed triplet construction method, N triplets (anchor, positive, negative) are constructed in a batch. These triplets originate from the hash codes generated by the cross-modal feature alignment network and will be used to calculate the triplet loss to optimize feature representation and enhance the model's discriminative ability. Specifically, it is expressed as follows:

[0017]

[0018] Wherein d(a i ,p i ) represents the Hamming distance between the anchor point and the positive sample, d(a i ,n i The distance () represents the Hamming distance between the anchor point and the negative sample. The margin is a hyperparameter used to control the distance difference between positive and negative sample pairs. When calculating the loss, only triplets that violate the margin are considered. This makes the model focus more on samples that are difficult to distinguish, ignoring easily distinguishable triplets, which helps reduce computational burden and makes the training process more efficient, thereby improving model performance and generalization ability. Given the non-differentiability of bit operations in the Hamming distance, an inner product is used to approximate the Hamming distance to ensure the differentiability of the loss function throughout the network training process, thus effectively optimizing feature representation and improving the model's alignment accuracy.

[0019] Furthermore, based on the characteristics of hash codes, an intermodal similarity loss is proposed, and the true similarity matrix S is constructed as follows:

[0020]

[0021] s ij Indicates two samples v i ,f i The true similarity is calculated as follows: if the labels of two samples are identical, the similarity is 1; otherwise, if the labels of two samples are inconsistent, the similarity is -1. ij =1, the elements at the same position in the hash code obtained by network mapping should be as similar as possible; otherwise, they should be as dissimilar as possible; that is, when s ij =1, network output and The similarity should ideally be close to 1; otherwise, it should be close to -1. Specifically, the predicted similarity matrix Y is calculated using the inner product of the two vectors:

[0022]

[0023] Where k is the hash code length, This represents the hash code of the i-th speech sample obtained through network mapping. Let represent the hash code of the j-th face sample obtained through mapping. The similarity loss is expressed as follows:

[0024]

[0025] In this context, ||·|| represents the distance between two points, using the Euclidean metric.

[0026] Overall loss function:

[0027]

[0028] Training: Since the feature extraction networks for each of the two modalities in the feature extraction stage use pre-trained networks, only the cross-modal feature alignment network is trained. A mini-batch training method is chosen, iterating through one batch of data at a time. The feature extraction network inputs features from two modalities into their respective feature extraction subnetworks to extract features. These extracted features are then used by the cross-modal feature alignment network to learn a compact hash code representation. Through multiple iterations—calculating the loss function, backpropagating, and updating the network parameters to minimize the loss function—the hash code output by the cross-modal feature alignment network ensures that voices and faces with the same identity are as close as possible in the hash space, while those with different identities are as far apart as possible. The hash codes for both modalities learned in each iteration are represented as follows:

[0029] B = [b1, b2, b3, ..., b n ] T ∈{+1,-1} n×k

[0030] Where B is the hash code matrix of speech or face obtained by network mapping, n is the size of the batch, representing the number of samples processed in each iteration; each element b in matrix B i Both voice and face samples are hash codes of length k, and each dimension of each hash code takes a value of +1 or -1.

[0031] The present invention proposes a cross-modal speech-face association technology based on hash learning, which can retrieve face pictures of the same identity from an image database according to an unknown speech for subsequent recognition tasks. In-depth studies in fields such as biology, cognitive science, and neuroscience have revealed the subtle connection between human voices and facial features. The vibration of the vocal cords and the facial structure of a person jointly act on the generation and transmission of sound, indicating that there is a certain correlation between the physiological / behavioral characteristics contained in the sound and facial features. This correlation provides a solid scientific basis for inferring facial features from sound; through experimental verification, the accuracy of the model of the present invention in cross-modal matching, retrieval, and verification tasks has been significantly improved compared with existing methods.

[0032] The present invention proposes a cross-modal deep residual alignment network based on hash learning. Using multiple linear layers and batch normalization layers, it can more effectively achieve the mapping from low-level features to high-level semantic features. On this basis, residual connections are used to fuse early layers with deep features, and the model can learn low-level detailed features and high-level abstract features, while eliminating the problem of gradient disappearance caused by the relatively deep network. During the network training process, the early layers are trained separately for the two modalities, and the later layers share weights for the two modalities. In this way, the multi-modal network can effectively utilize the unique information of each modality, while achieving cross-modal information integration and feature learning. Finally, a hash function is added to the last layer of the network to map the features into a compact hash code form, greatly improving the computational efficiency.

[0033] The present invention adopts a new method for constructing triplets. The present invention selects a more balanced strategy, that is, selects sample pairs where the distance between the anchor point and the positive sample is less than the distance between the anchor point and the negative sample, and this distance difference is less than a preset boundary value (margin). Specifically, the present invention selects negative samples that satisfy the condition 0 < d(a i ,n i ) - d(a i ,p i ) < margin. In this way, the present invention ensures that the negative samples in the triplets are neither too easy to distinguish nor too difficult, avoiding the collapse phenomenon during the network training process, that is, preventing the model from falling into local minima, thereby promoting the model to learn the distance metric more evenly in the feature space. Through experimental verification, the accuracy of the model of the present invention in cross-modal matching, retrieval, and verification tasks has been significantly improved compared with existing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is the overall network framework diagram of the present invention.

[0035] Figure 2 is the t-SNE visualization diagram of the hash code of the present invention, Figure 2 (a) is the feature distribution map before learning. Figure 2 (b) is the feature distribution map after learning. Detailed Implementation

[0036] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0037] like Figure 1 As shown, a cross-modal speech-face association method based on heterogeneous hash networks is presented, with the following specific steps:

[0038] Dataset Construction: The network was trained and tested using the pre-processed VoxCeleb dataset. This voice-face dataset includes voice clips and face images for 1225 identities, represented as follows: Where i represents the i-th identity, and N = 1225. Each identity is internally represented as follows: in, Let j represent the j-th speech segment of the i-th identity, where j ranges from 1 to m, meaning the i-th identity has a total of m speech samples; similarly, The i-th identity has a j-th face image frame, where j ranges from 1 to n, meaning there are n face image frames for the i-th identity. To ensure a fair comparison, the frames are divided into a training set, a test set, and a validation set, following the partitioning method used in previous work.

[0039] Loading the dataset: Since triplet loss is needed for backpropagation to optimize network parameters, a sufficient number of triples (anchor, positive, negative) need to be selected in each iteration. Therefore, a sampler is added to ensure that the model receives diverse samples during training. Each iteration contains samples with multiple different identities, and each identity has a fixed number of instances, in order to select more effective triples for subsequent triplet loss calculation. During network training, the data is presented as a tuple in each iteration, specifically... Where v represents a speech segment, f represents a face image, and l is a label, if l i =l j , indicating v i ,f i With v j ,l j Belonging to the same person, if l i ≠l j These are people with different identities, and batch represents the number of tuples in each iteration of the network.

[0040] Construct a speech feature extraction network A v (·): The ECAPA-TDNN network, pre-trained on the VoxCeleb2 dataset, is used as the speech feature extraction network. For a speech segment, its Mel-spectral features are first extracted and normalized, and then used as the input of the ECAPA-TDNN network to extract features. The output is a 192-dimensional vector as the final speech feature.

[0041] Construct a face feature extraction network A f (·): The Inception-V1 network, pre-trained on the VGGFace2 dataset, is used as the face feature extraction network. The face image is processed by the network and outputs a 512-dimensional vector as the final face feature.

[0042] A cross-modal feature alignment network φ(·,θ)={φv(·,θv),φf(·,θf)} is constructed: the features of the two modalities obtained in the feature extraction stage are used as inputs to this network, and the network automatically inputs the features of the two modalities into their corresponding feature alignment sub-networks. In terms of network structure design, multiple fully connected layers FC(·) are used to achieve feature alignment. To reduce internal covariate shift, except for the last fully connected layer, each fully connected layer is followed by a batch normalization layer BN(·). Batch normalization can be represented as follows:

[0043]

[0044] Where, x i μ represents the features after passing through the fully connected layer. B σ represents the mean of all sample features in a batch across each feature channel. Bε represents the variance of all sample features in each feature channel within a batch. ε is a small constant used for numerical stability to prevent division by zero. γ and β are learnable parameters used to scale and translate the normalized features to restore the model's expressive power and allow the network to learn the optimal representation of the normalized features. Simultaneously, the batch-normalized features of the first layer are residually connected to the features of the last layer, fusing early and deep features, allowing effective integration of features captured by the early layers with high-level features learned by the deep layers. The two modalities in the early layers are trained independently, while the later layers share weights for both modalities, enabling the cross-modal network to learn more general feature representations. This network maps the original 512-dimensional face features and 192-dimensional speech features to a 128-dimensional space. A hash function layer sgn(·) is added to the last layer of the network, causing the network to output a fixed-length hash code. Here, sgn(·) is a sign function that returns 1 if the element is positive and -1 otherwise. The learned hash code representation enables voices and faces with the same identity to have a shorter Hamming distance in the hash space, while voices and faces with different identities have a longer distance. The final output of the network is:

[0045] V = sgn(φ v (A v (v i ),θ v )

[0046] F = sgn(φ) f (A f (f i ),θ f )

[0047] Where, φ v φ represents the speech feature alignment subnetwork in the constructed cross-modal feature alignment network. f A represents the face feature alignment subnetwork in the constructed cross-modal feature alignment network. v (v i A represents the speech features obtained through a feature extraction network. f (f i θ represents the facial features obtained through a feature extraction network. v θ f F and V represent the parameters learned by the two sub-networks in the cross-modal feature alignment network, respectively, and F and V are the hash codes of the two modalities obtained in the end.

[0048] Constructing the loss function: Based on the proposed triplet construction method, N triplets (anchor, positive, negative) are constructed in a batch. These triplets originate from the hash codes generated by the cross-modal feature alignment network and will be used to calculate the triplet loss to optimize feature representation and enhance the model's discriminative ability. Specifically, it is expressed as follows:

[0049]

[0050] Wherein d(a i ,p i ) represents the Hamming distance between the anchor point and the positive sample, d(a i ,n i The distance () represents the Hamming distance between the anchor point and the negative sample. The margin is a hyperparameter used to control the distance difference between positive and negative sample pairs. When calculating the loss, only triplets that violate the margin are considered. This makes the model focus more on samples that are difficult to distinguish, ignoring easily distinguishable triplets, which helps reduce computational burden and makes the training process more efficient, thereby improving model performance and generalization ability. Given the non-differentiability of bit operations in the Hamming distance, an inner product is used to approximate the Hamming distance to ensure the differentiability of the loss function throughout the network training process, thus effectively optimizing feature representation and improving the model's alignment accuracy.

[0051] Furthermore, based on the characteristics of hash codes, an intermodal similarity loss is proposed, and the true similarity matrix S is constructed as follows:

[0052]

[0053] s ij Indicates two samples v i ,f i The true similarity is calculated as follows: if the labels of two samples are identical, the similarity is 1; otherwise, if the labels of two samples are inconsistent, the similarity is -1. ij =1, the elements at the same position in the hash code obtained by network mapping should be as similar as possible; otherwise, they should be as dissimilar as possible; that is, when s ij =1, network output and The similarity should ideally be close to 1; otherwise, it should be close to -1. Specifically, the predicted similarity matrix Y is calculated using the inner product of the two vectors:

[0054]

[0055] Where k is the hash code length, This represents the hash code of the i-th speech sample obtained through network mapping. Let represent the hash code of the j-th face sample obtained from the mapping. The similarity loss is expressed as follows:

[0056]

[0057] In this context, ||·|| represents the distance between two points, using the Euclidean metric.

[0058] Overall loss function:

[0059]

[0060] Training: Since the feature extraction networks for each of the two modalities in the feature extraction stage use pre-trained networks, only the cross-modal feature alignment network is trained. A mini-batch training method is chosen, iterating through one batch of data at a time. The feature extraction network inputs features from two modalities into their respective feature extraction subnetworks to extract features. These extracted features are then used by the cross-modal feature alignment network to learn a compact hash code representation. After multiple iterations—calculating the loss function, backpropagating, and updating the network parameters—to minimize the loss function, the hash code output by the cross-modal feature alignment network ensures that voices and faces with the same identity are as close as possible in the hash space, while those with different identities are as far apart as possible. The hash codes for both modalities learned in each iteration are represented as follows:

[0061] B = [b1, b2, b3, ..., b n ] T ∈{+1,-1} n×k

[0062] Where B is the hash code matrix of speech or face obtained from network mapping, and n is the batch size, representing the number of samples processed in each iteration. Each element b in matrix B i Both voice and face samples are hash codes of length k, and each dimension of each hash code takes a value of +1 or -1.

[0063] All experiments in this invention were conducted on an NVIDIA GeForce RTX 4090 graphics card, using the PyTorch deep learning framework and Python 3.9 as the development foundation. During model training, the Adam optimizer was employed with a learning rate of 1e-3 and a margin parameter of 1 to optimize the model's learning ability and generalization performance. Finally, the Voxceleb dataset was used, divided into training, validation, and test sets, for training, and the network's performance was tested on cross-modal matching, retrieval, and validation tasks.

[0064] I. Cross-modal 1:2 matching: Select one sample in the speech modality and two samples in the face image modality to find face images belonging to the same identity as the speech sample; conversely, use the face image modality as the query to find speech segments belonging to the same identity as the face image sample. The performance of the model is verified by calculating the matching accuracy. As shown in Table 1, the accuracy of the present invention is significantly improved compared with the results of other researchers.

[0065] Table 1. Accuracy of cross-modal 1:2 matching task

[0066] method VF FV SVHF 81.00 79.50 FV-CME 78.10 77.80 LAFV 78.20 78.60 PINs 83.80 DIMNet-I 83.45 83.52 LDJE 85.18 85.42 0urs 85.58 85.68

[0067] II. Cross-modal verification: Given a speech sample and a face sample, determine whether they belong to the same person. This task uses the area under the curve (AUC) as the evaluation metric, and the comparison with previous work is shown in Table 2.

[0068] Table 2. AUC values ​​for cross-modal validation tasks

[0069] method AUC PINs 78.50 SSNet 78.80 DIMNet-I 82.50 Bi-Pcm-FST 85.00 Ours 86.93

[0070] III. Cross-modal retrieval: Given a test sample of one modality, it is necessary to query positive samples that match the test sample from the total dataset. This task will use mean average precision (mAP) as the evaluation metric, and the comparison with previous work is shown in Table 3.

[0071] Table 3 Performance of mAP in cross-modal retrieval

[0072] method VF FV FV-CME 1.96 2.18 VFMR3 5.00 DIMNet-I 4.25 4.17 Bi-Pcm-FST 6.36 6.04 0urs 7.58 7.98

[0073] IV. Ablation Experiments: This invention compares the performance of networks with only two layers and multiple layers on cross-modal matching, verification, and retrieval tasks. In addition, it compares the performance of networks using cross-modal similarity loss and those without using this loss, as shown in Table 4.

[0074] Table 4 Evaluation of different network depths on cross-modal matching, validation, and retrieval tasks.

[0075]

[0076] Previous studies used floating-point numbers to align samples for features and measured similarity between samples by calculating cosine similarity and Euclidean distance. Smaller distances between samples indicated higher similarity, while larger distances indicated lower similarity. This invention proposes mapping samples to fixed-length hash codes. Due to the binary nature of hash codes, their similarity can be quickly assessed by calculating the inner product of the hash codes of two samples. Furthermore, hash codes can be represented using half-precision floating-point numbers, which simplifies storage and significantly reduces memory usage. By using hash codes, this invention significantly reduces the time and space complexity of the algorithm, thereby improving its efficiency. For example, for two 256*128 speech and face feature matrices, representing the similarity between samples using cosine similarity requires approximately 5.01e+7 computational operations, while this invention can represent the similarity between samples using the inner product, requiring approximately 1.61e+7 computational operations.

[0077] V. Visualization of online learning: such as Figure 2 As shown, this invention further utilizes the t-SNE algorithm to perform 2D visualization of the learned high-dimensional face-speech embedding features, where data points of the same color come from the same identity sample. Figure 2 (a) shows the feature distribution before training. It can be clearly seen that, due to the heterogeneity of the two modal data, the same color is obviously divided into two parts: one part is speech samples and the other part is face samples, and the distribution is relatively scattered. Figure 2 (b) shows the results after training by the method of the present invention. The present invention not only eliminates the heterogeneity between modalities, but also makes two modal samples with the same identity closer to each other, while samples with different identities are far apart from each other. This shows that the heterogeneous deep residual hash alignment network proposed in the present invention can effectively learn the correlation between modalities.

[0078] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included within the scope of the present invention.

Claims

1. A method for cross-modal speech and face association based on heterogeneous hash networks, characterized in that, The specific steps are as follows: Step 1: Build the dataset: Use the pre-processed dataset to train and test the network; Step 2: Load the dataset: Add a sampler to receive samples during training. Each iteration contains samples with multiple different identities, and each identity contains a fixed number of instances. During network training, the data is presented as a tuple in each iteration, as shown below: ; in, Indicates a speech segment, Represents a human face image. For tags, if ,express and Belonging to the same person; if Then it is determined and People belonging to different identities; batch represents the number of tuples in each iteration of the network. Step 3: Establish a speech feature extraction network The ECAPA-TDNN model, pre-trained on the VoxCeleb2 dataset, is used as the speech feature extraction network. For a speech segment, the Mel spectrum features of the speech are first extracted and normalized, and then used as the input of the ECAPA-TDNN model to extract features, and the final speech features are output. Step 4: Construct a facial feature extraction network The Inception-V1 network, pre-trained on the VGGFace2 dataset, is used as the face feature extraction network. The face image is processed by the Inception-V1 network to output a 512-dimensional vector as the final face feature. Step 5: Construct a cross-modal feature alignment network: The specific expression for the cross-modal feature alignment network is as follows: ; The speech and facial features obtained in the feature extraction stage are used as inputs to this network. The cross-modal feature alignment network automatically inputs the speech and facial features into the corresponding feature alignment sub-networks, using multiple fully connected layers. To achieve feature alignment, except for the last fully connected layer, each fully connected layer is immediately followed by a batch normalization layer. The batch normalization representation is as follows: ; in, This represents the features after passing through the fully connected layer. This represents the mean of all sample features in a batch across each feature channel. This represents the variance of all sample features within a batch in each feature channel. It is a small constant. and These are learnable parameters; Add a hash function layer to the last layer of the cross-modal feature alignment network. The final output of the cross-modal feature alignment network is: ; ; in, , This represents the speech feature alignment subnetwork in the constructed cross-modal feature alignment network. This represents the face feature alignment subnetwork in the constructed cross-modal feature alignment network. The speech features are obtained through a feature extraction network. The facial features obtained through a feature extraction network, These represent the parameters learned by the two subnetworks in the cross-modal feature alignment network. This refers to the hash codes of the two modalities obtained at the end.

2. The method for cross-modal speech and face association based on heterogeneous hash networks according to claim 1, characterized in that, In step one, the dataset includes voice clips and facial images of 1225 identities, and the dataset is represented as follows: Where i represents the i-th person, and N=1225; Each identity is internally represented as ,in, Let j represent the j-th speech segment of the i-th identity, where j ranges from 1 to m, meaning the i-th identity has a total of m speech samples; similarly, The j-th face image frame represents the i-th identity, where j ranges from 1 to n, meaning there are a total of n face image frames for the i-th identity.

3. The method for cross-modal speech and face association based on heterogeneous hash networks according to claim 2, characterized in that, In step five, the batch-normalized features of the first layer of the network are residually connected with the features of the last layer, fusing the features of the early layer and the deep layer. The two modalities in the early layer are trained separately, and the weights of the two modalities are shared in the later layers. Through this network, the original 512-dimensional face features and 192-dimensional speech features are mapped to a 128-dimensional space.

4. The method for cross-modal speech and face association based on heterogeneous hash networks according to claim 3, characterized in that, A loss function is constructed by generating N triples in a batch according to the triple construction method. These N triples originate from hash codes generated by the cross-modal feature alignment network and are used to calculate the triple loss, thereby optimizing feature representation and enhancing the model's discriminative ability. ; in, This represents the Hamming distance between the anchor point and the positive sample. The distance between the anchor point and the negative sample is represented by the Hamming distance, and margin is a hyperparameter used to control the distance difference between positive and negative sample pairs. Furthermore, based on the characteristics of hash codes, an intermodal similarity loss is proposed, and the true similarity matrix S is constructed as follows: ; in, Representing two samples The true similarity is calculated as follows: if two samples have the same label, the similarity is 1; if the two samples have different labels, the similarity is -1. The predicted similarity matrix Y is calculated using the inner product of the two vectors. ; in, The length of the hash code. This represents the first result obtained through network mapping. The hash code of a voice sample. The mapping results in the first... The hash code of an individual face sample represents the similarity loss as follows: ; in, It is used to represent the distance between two points, and the Euclidean metric method is used. The overall loss function is: 。 5. The method for cross-modal speech and face association based on heterogeneous hash networks according to claim 4, characterized in that, The cross-modal feature alignment network is trained using a mini-batch training method, with each iteration using one batch of data. The specific implementation is as follows: ; The feature extraction network inputs features from two modalities into their respective feature extraction subnetworks to extract features. The extracted features are used to learn hash code representations for the cross-modal feature alignment network. After multiple iterations—that is, after calculating the loss function, backpropagation is performed to update the network parameters to minimize the loss function—the hash code representations of the two modalities learned in each iteration are as follows: ; Where B is the hash code matrix of speech or face obtained by network mapping, n is the batch size, representing the number of samples processed in each iteration; each element in matrix B Both voice samples and face samples are hash codes of length k, and each dimension of each hash code takes a value of +1 or -1.

Images