Audio-visual matching method based on relation awareness attention correction network

Through the parallel module and feature orthogonal constraints of the relationship perception correction attention network, combined with the adversarial network and relative distance stretch measurement loss, the problem of subtle feature adjustment and interference term impact in audio-visual matching is solved, and more accurate and resilient Cross-modal feature matching of rods.

CN118897902AInactive Publication Date: 2024-11-05WUHU SIMBA NETWORK TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202410972196.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-05
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The prior art is difficult to finely adjust subtle features in audio-visual matching, resulting in the model paying too much attention to some attribute features or neglecting key identity features. Interference items such as background noise and multi-person conversations affect the accuracy of audio identity information, reducing the generalization of the model. ability.

Method used

Using a relationship perception correction attention network, cross-modal features are performed through parallel adaptive modal correction attention modules and relationship perception modal correction attention modules, combining feature orthogonal constraints and dynamic residual weight combinations. The optimal matching of the adversarial network and relative distance stretch measurement loss are introduced to enhance the robustness of audio-visual matching.

Benefits of technology

Effectively explore the intrinsic connections of cross-modal semantic features, eliminate the pseudo-correlation of interfering information, improve the accuracy and robustness of audio-visual matching, and achieve more accurate cross-modal feature matching.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118897902A_ABST
    Figure CN118897902A_ABST
Patent Text Reader

Abstract

The invention discloses an audiovisual matching method based on a relation perception attention correction network. Internal association between semantic features of different modes is explored by utilizing correction attention between relation perception modes and correction attention in a self-adaptive mode. The correction attention mechanism between the relation perception modes guides features of different modes to perform target-specific interaction through correlation information obtained by a relation perception network so as to identify potential semantic relations. In view of significant information difference possibly existing between different modes, correction of attention in the adaptive mode serves as an auxiliary mechanism, the features can be independently and effectively processed, and then feature representation of each mode is deeply mined. In addition, the two attention modules are integrated with self-adaptive attention correction units, so that attention distribution is optimized, feature representation is enhanced, and meanwhile, the influence of interference information on the network is reduced. By enhancing the contrast between positive and negative samples, the relative distance stretch metric loss helps to improve the embedded characterization of the audiovisual features. According to the method, testing is carried out on a plurality of public reference data sets, various scenes are covered, and good performance is shown.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to computer vision and cross-modal biological information matching technology, and in particular to an audio-visual matching method based on a relation-aware corrected attention network. Background Art

[0002] Neuroscientists have confirmed that both voices and faces contain unique identity information and are suitable for recognition. Therefore, people often naturally associate faces with voices. For example, just by hearing a familiar voice, we can match it with the corresponding face, which is called "recognizing people by hearing the voice." This has led to the rise of a new research field (audio-visual matching), which aims to study how different biometric modalities interact with each other and use neuroimaging to provide a scientific basis for these interactions. Audio-visual matching technology is widely used in many research and application scenarios such as criminal investigation security, intelligent surveillance systems, and intelligent voice recording. A major challenge in this field is to mimic the human ability to perceive the correlation of multimodal biometric signals. This involves extracting features from facial images and audio clips to establish connections between cross-modal identity features.

[0003] Previous studies have shown that adversarial training can significantly reduce the heterogeneity problem between modalities, while subtle feature extraction can help reduce the impact of identity differences. Nevertheless, such methods may inadvertently capture irrelevant features, thereby impairing the generalization ability of the model. In addition, in interviews, background noise, the content of the interviewee's speech, and the superimposed sounds of multiple people talking can reduce the accuracy of audio identity information. These interference items can lead to overfitting of global features under a single modality, further increasing the complexity of the audio-visual matching task.

[0004] To solve this problem, one method is to use modal decomposition to distinguish identity-related features from modality-related features, and process these features separately to learn effective feature representations and filter out interfering features. Another method is to randomly discard keys before calculating the attention matrix to reduce the overfitting of the model to interfering features. These methods all require adjusting parameters based on experience and specific tasks to achieve optimal performance. Therefore, the use of a soft threshold layer can be used as an adaptive disentanglement method to achieve the purpose of feature decomposition.

[0005] However, as a disentanglement technique, the soft threshold layer fails to completely cut off the connection between identity features and interference features due to the lack of effective separation constraints. To address this problem, two strategies are usually adopted: one is attribute decomposition, which uses attribute-oriented strategies to identify the hierarchical relationship between different attribute features; the other is data decomposition, which uses dynamic weighting strategies to adjust the sample importance of face images and sound clips to avoid overfitting the network to samples containing a large amount of disturbance information.

[0006] However, both of the above methods fail to fine-tune subtle features, which may lead to focusing only on some attribute features or ignoring key identity representation features. Therefore, it is necessary to design an effective intervention strategy to assist the soft threshold layer to achieve effective feature decomposition, so as to achieve effective audio-visual modality representation feature matching. Summary of the invention

[0007] Purpose of the invention: The purpose of the present invention is to solve the deficiencies in the prior art and to provide an audio-visual matching method based on a relation-aware corrected attention network.

[0008] Technical solution: The present invention provides an audio-visual matching method based on a relation-aware corrected attention network, comprising the following steps:

[0009] Step 1: Get an anchor audio clip And the corresponding k face images The anchor audio is used as the recognition item and the face image is used as the gallery matching item; then the features of the anchor audio segment and the face image are extracted respectively to obtain the original audio segment features f i a and facial image features i represents the i-th data tuple; k>1 represents a matching situation, and k=1 represents a special matching situation;

[0010] Step 2: Transform the audio clip feature f i a and features of face images The audio clips are sent to the relation-aware corrected attention network, and the similarity between the audio clips and the single face image is calculated through the relation-aware network to obtain their respective attention matrices. and The relation-aware corrective attention network includes a parallel adaptive intra-modality corrective attention module RIRA and a relation-aware inter-modality corrective attention module AIRA;

[0011] The relation-aware inter-modal correction attention module AIRA uses the adaptive attention correction unit AARU to and Adjust the attention matrix and Then, the two attention matrices are subjected to first-order inter-modal interaction and second-order inter-modal interaction in turn, and finally the feature and

[0012] The adaptive intra-modality correction attention module RIRA uses the adaptive attention correction unit AARU to and Correction is performed to obtain the corrected attention matrix within the modality and Then respectively and Use decoder to extract relevant modality features and

[0013] Step 3: First introduce the feature orthogonality constraint L orth , so that the features unrelated to the modality and the features related to the modality complement each other, ensuring the orthogonality of the features between the two modalities; using the dynamic residual weight combination method, combined with the dynamically selected modal internal and inter-modal enhanced features and global features, the optimal matching between cross-modal features is performed, and the features of the two modalities with the best matching are recorded as and

[0014] Step 4: Utilize audio features through adversarial network discriminator and facial image features To generate modality-independent features

[0015] Furthermore, step 1 also calculates the correlation estimate between the modes pass Evaluate the semantic relevance between different modalities. The calculation formula is as follows:

[0016]

[0017] The above formula is obtained by conv s Convolutional layers map the semantics of audio and visual images into the same shared space;

[0018] Next, Used as a pseudo-label, further supervised learning is performed by assigning its true label and calculating the relationship-aware loss L RA as follows:

[0019]

[0020] Among them, "epoch" represents the current iteration number, and "N" represents the total iteration number. σ represents the sigmoid activation function; the temperature control parameter is set to 5; the directional guided interaction depends more on the positive or negative correlation rather than its specific numerical value, so a simulated annealing weight is introduced to reduce the impact of loss on the model.

[0021] In order to focus on the relevant semantics in cross-modal interaction and use context-aware relevance to guide such interaction, the present invention introduces an inter-modal correction attention mechanism. The specific working content of the relationship-aware inter-modal correction attention module AIRA is:

[0022] First, for the similarity between the audio clip and the face image, the attention matrix is ​​obtained and The expression is as follows:

[0023]

[0024] In the above formula, and They refer to the attention matrices of images and audio respectively;

[0025] Then, the adaptive attention correction unit AARU is used to learn the cross-modal semantic threshold to reduce the influence of interference information, and the attention matrix is ​​adjusted to obtain the adjusted attention matrix and The expression is as follows:

[0026]

[0027]

[0028] In the above formula, ⊙ represents the product of elements, clamp represents the truncation operation, and sign represents the sign function, which is defined as follows:

[0029]

[0030] Next, the first-order modal interaction between audio and face image is performed, and the expression is as follows:

[0031]

[0032] and They represent the audio clip features and the k-th face image features in the first-order inter-modal interaction, respectively. Due to the heterogeneity between modalities, it is difficult to fully explore the cross-modal semantic connections only through the first-order inter-modal interaction;

[0033] Furthermore, the hyperparameter δ is used to sparse the similarity matrix of the first-order inter-modal interaction to identify the closest local semantic association, and the semantic connection is enhanced through the intra-modal attention mechanism; finally, features are extracted through the second-order attention interaction, and the expression is as follows:

[0034]

[0035] In the above formula, decode s It is a decoding convolutional layer used to restore the features shared by audio and face images to the original feature dimension. and Diffusion of characteristic areas of interest.

[0036] In order to perform modality alignment, adaptive intra-modality attention is introduced here. The working content of the adaptive intra-modality correction attention module RIRA is as follows:

[0037] In order to reduce the background interference within the modality and strengthen the representation of foreground features, the attention matrix after intra-modality correction is first obtained and The expression is as follows:

[0038]

[0039]

[0040] Among them, conv p is a dimensionality reduction convolution operation, ε is a positive threshold for online learning; for the matrix and The positive elements less than ε are set to zero by the clamp function, and the negative elements greater than -ε are set to zero by the clamp function;

[0041] Then, decode them separately and get the following features:

[0042]

[0043] Among them, σ is the constant term mentioned before, which is set to 10 in the experiment; decode p It is a dimensionality-increasing convolutional layer used to restore modality-related features to the original extraction dimension.

[0044] Furthermore, the feature orthogonality constraint L in step 3 orth The expression is as follows:

[0045]

[0046] Among them, G is the maximum pooling. M represents the number of small batch data tuples.

[0047] In order to adjust the global characteristics between the two modalities and the enhanced semantics inside and outside the modalities, the optimal matching features are obtained through a dynamic residual weight combination strategy, which is expressed as follows:

[0048]

[0049] Among them, m s represents the feature weight associated with the modality, m p Represents the feature weight that is independent of the modality; these two weights are calculated by SENet; in order to reduce the variance of the modal features, instance normalization (IN) is used.

[0050] Furthermore, the step 4 generates modality-independent features through the discriminator Finally, the nonlinear discrimination ability of the multilayer perceptron is used to identify potential matches, and the cross entropy method is used to calculate the loss value;

[0051]

[0052] Among them C m Indicates matching classification; l i is the matching identity tag;

[0053] In order to enhance the robustness of audio-visual matching and improve the performance of identity association recognition, the relative distance stretching metric loss is used to expand the distance between identity-related features and reduce the interval between non-identity-related features. The cross-modal relative distance stretching metric loss is as follows:

[0054]

[0055] The hyperparameter μ 1 and μ 2 are set to 10 and 4 respectively, and the parameter θ is set to 1.2; Represents anchor samples With positive samples The Euclidean distance between is the anchor sample With negative samples The Euclidean distance between It is a positive sample With negative samples The Euclidean distance between Indicates whether the identities match, and if not 1, when matching is 0; label Helps calculate the distance between each negative sample in the candidate sample. To achieve the relative distance stretching constraint, select the negative sample closest to the positive sample from many candidate negative samples and emphasize their distance.

[0056] Furthermore, the total loss is:

[0057] L=L cls +L dis +L gen +L RDSM ,

[0058] L cls Calculate the loss for cross entropy, L RDSM Stretch metric loss for cross-modal relative distance.

[0059] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0060] (1) The relation-aware rectified attention network proposed in this paper can effectively explore the intrinsic connections between cross-modal semantic features. This capability is achieved through the synergy of relation-aware inter-modal rectified attention and adaptive intra-modal rectified attention designed as a parallel structure.

[0061] (2) The adaptive attention correction unit of the present invention can eliminate the pseudo-correlation caused by interference information and guide the mutual enhancement of cross-modal semantic features.

[0062] (3) The relative distance stretching metric loss in the present invention utilizes the distance relationship between anchor points, positive samples and negative samples to push mismatched features further apart, thereby promoting robust audio-visual matching embedding representation. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0064] Figure 2 It is a schematic diagram of the network model of the present invention;

[0065] Figure 3 It is a schematic diagram of the dynamic residual weight combination in the present invention;

[0066] Figure 4 The quantitative comparison results of the embodiments with respect to the prior art are compared. DETAILED DESCRIPTION

[0067] The technical solution of the present invention is described in detail below, but the protection scope of the present invention is not limited to the embodiments.

[0068] like Figure 1 As shown, the audio-visual matching method based on the relation-aware correction attention network of the present invention comprises the following steps:

[0069] Step 1: Get an anchor audio clip And the corresponding k face images The anchor audio is used as the recognition item and the face image is used as the gallery matching item; then the features of the anchor audio segment and the face image are extracted respectively to obtain the original audio segment features f i a and facial image features i represents the i-th data tuple;

[0070] Step 2: Transform the audio clip feature f i a and features of face images The relationship-aware rectified attention network calculates the similarity between the audio clip and a single face image to obtain their respective attention matrices. and The relation-aware corrective attention network includes a parallel adaptive intra-modality corrective attention module RIRA and a relation-aware inter-modality corrective attention module AIRA;

[0071] The relation-aware inter-modal correction attention module AIRA uses the adaptive attention correction unit AARU to and Adjust the attention matrix and Then, the two attention matrices are subjected to first-order inter-modal interaction and second-order inter-modal interaction in turn, and finally the feature and

[0072] The adaptive intra-modality correction attention module RIRA uses the adaptive attention correction unit AARU to and Correction is performed to obtain the corrected attention matrix within the modality and Then respectively and Use decoder to extract relevant modality features and

[0073] Step 3: First introduce the feature orthogonality constraint L orth , so that the features unrelated to the modality and the features related to the modality complement each other; the dynamic residual weight combination method is used to combine the dynamically selected modality internal and inter-modality enhanced features with the global features to perform the optimal matching between cross-modality features. The features of the two modalities with the best matching are recorded as and

[0074] Step 4: Since the present invention is ultimately to generate features that are independent of the modality, due to the heterogeneity between the two modalities, the influence cannot be completely eliminated only through interaction and feature embedding, which causes the learned feature differences to interfere with the final discrimination result; the present invention uses the audio features through the adversarial network discriminator and facial image features To generate modality-independent features Reduce the impact of modal heterogeneity.

[0075] In this embodiment, the data can be divided into a training set, a validation set, and a test set, and these data are applied according to the characteristics of their respective tasks. The image data is resized to a size of 3×224×244 and input into the ResNet network for feature extraction, and the model parameters pre-trained on the ImageNet dataset are used. We use the Python module Librosa to analyze audio signals and extract features through Mel frequency cepstral coefficients (MFCC). MFCC is a set of about 10-20 features that generally describes the envelope shape of the spectrum. MFCC simulates the characteristic waveform of human voice and further extracts high-level semantic information through the ResNet network. Finally, the corresponding modal semantic representation is extracted from the original audio and image information.

[0076] like Figure 2 As shown, both the inter-modal correction attention mechanism and the adaptive intra-modal correction attention mechanism in the present invention are to effectively focus on the relevant semantics in cross-modal interaction and use context-aware correlation to guide such interaction. Evaluate the semantic relevance between different modalities. The calculation formula is as follows:

[0077]

[0078] The above formula is obtained by conv s Convolutional layers map the semantics of audio and visual images into the same shared space;

[0079] Next, Used as a pseudo-label, further supervised learning is performed by assigning its true label and calculating the relationship-aware loss L RA as follows:

[0080]

[0081] Among them, epoch represents the current iteration number, N represents the total iteration number, and δ represents the sigmoid activation function. The pseudo label obtained It is also used to guide the interaction of cross-modal features during testing.

[0082] In the audio-visual matching method based on the relation-aware corrective attention network of this embodiment, the specific working content of the relation-aware inter-modal corrective attention module AIRA is as follows:

[0083] First, for the similarity between the audio clip and the face image, the attention matrix is ​​obtained and The expression is as follows:

[0084]

[0085] In the above formula, and They refer to the attention matrices of images and audio respectively;

[0086] Then, the attention matrix is ​​adjusted by learning the cross-modal semantic threshold through the adaptive attention correction unit AARU (reducing the influence of interference information and improving the accuracy of audio-visual matching) to reduce the influence of interference information, and the adjusted attention matrix is ​​obtained. and The expression is as follows:

[0087]

[0088] In the above formula, ⊙ represents the product of elements, clamp represents the truncation operation, and sign represents the sign function, which is defined as follows:

[0089]

[0090] Since the correlation between audio and face images is based on all related segments, the correlation of each segment is adjusted and the attention map is updated to focus on learning the correlation between cross-modal semantics. This process is called first-order inter-modal interaction. The expression of the first-order inter-modal interaction between audio and face images is as follows:

[0091]

[0092] Furthermore, the similarity matrix of the first-order inter-modal interaction is sparsely processed using the hyperparameter δ, and the semantic connection is enhanced through the intra-modal attention mechanism; finally, features are extracted through the second-order attention interaction, and the expression is as follows:

[0093]

[0094] In the above formula, Represents the features of the k-th face image, decode s is a decoding convolutional layer used to restore the features shared by audio and face images to the original feature dimension, while the hyperparameter δ performs sparse processing on the similarity matrix of the first-order modal interactions to more accurately locate closely related local semantic regions.

[0095] In order to meet the challenge of significant information differences between audio-visual modalities, eliminate the influence of the external environment on the difference between audio and face image information, achieve cross-modal interaction and alignment, and obtain semantically rich modality-independent feature representation, this implementation uses the adaptive intra-modality correction attention module RIRA for effective feature representation. The work content is as follows:

[0096] First get the attention matrix after intra-modal correction and The expression is as follows:

[0097]

[0098] Among them, conv p is a dimensionality reduction convolution operation, ε is a positive threshold for online learning; for the matrix and The positive elements less than ε are set to zero by the clamp function, and the negative elements greater than -ε are set to zero by the clamp function;

[0099] Then, decode them separately and get the following features:

[0100]

[0101] Among them, δ is a constant term, decode p It is a dimensionality-enhancing convolutional layer.

[0102] To ensure that the modality-independent features are complementary to the relevant features, the feature orthogonality constraint L is used. orth , the expression is as follows:

[0103]

[0104] Among them, G is the maximum pooling; M represents the number of small batch data tuples

[0105] like Figure 3 As shown, since the cross-modal matching effect in the matching process strongly depends on the global contextual correlation between the modalities, this correlation is determined by the global and local enhanced features, which are indirectly extracted from each modality. Therefore, it is crucial to adjust the global features between different modalities, and the enhanced semantics within and between modalities to establish a close correlation. The optimal matching feature expression obtained by the dynamic residual weight combination strategy in this embodiment is as follows:

[0106]

[0107] Among them, m s represents the feature weight associated with the modality, m p represents the feature weights that are independent of the modality.

[0108] Since interaction cannot fundamentally solve the heterogeneity problem between cross-modal features, this embodiment introduces the adversarial idea, combining the discriminator with the audio feature f i a and facial image features To generate modality-independent features In this process, the nonlinear discrimination ability of the multilayer perceptron is used to identify potential matches, and the cross entropy method is used to calculate the loss value;

[0109]

[0110] Among them C m Indicates matching classification; l i is the matching identity tag.

[0111] And use the relative distance stretching metric loss to expand the distance between identity-related features and narrow the interval between non-identity-related features. The cross-modal relative distance stretching metric loss is as follows:

[0112]

[0113] The hyperparameter μ 1 and μ 2 are set to 10 and 4 respectively, and the parameter θ is set to 1.2; Represents anchor samples With positive samples The Euclidean distance between is the anchor sample With negative samples The Euclidean distance between It is a positive sample With negative samples The Euclidean distance between Indicates whether the identities match, if not 1 if it matches is 0.

[0114] The total training loss of the present invention is:

[0115] L=L cls +L dis +L gen +L RDSM ,

[0116] L cls Calculate the loss for cross entropy, L RDSM is the cross-modal relative distance stretching metric loss. During training, the shared layer is updated by all training data, while the fully connected layer of each branch is iteratively updated according to the corresponding modality features.

[0117] Depend on Figures 1 to 3 It can be seen that the entire relation-aware corrected attention network of the present invention includes relation-aware corrected attention, adaptive intra-modal corrected attention, and relative distance stretching metric loss. The relation-aware network enhances the interaction of cross-modal semantic features through correlation guidance, eliminates pseudo-correlation in interference information, and restricts the distribution of samples of different identities, so that the model can learn stable cross-modal representation features to achieve more accurate audio-visual cross-modal matching.

[0118] Example

[0119] In order to further verify the performance of the technical solution of the present invention, this implementation uses a public data set to compare the present invention with the prior art.

[0120] The VoxCeleb dataset is a public audio dataset that contains more than 149,354 audio clips involving 1,225 different speakers. The VGGFace dataset is a public face image dataset that contains 137,060 facial images extracted from the same video. In this embodiment, individuals whose names begin with "A" or "B" are assigned to the validation set, and individuals whose names begin with "C", "D" or "E" are assigned to the test set. Individuals whose first letters are from "F" to "Z" are assigned to the training set.

[0121] The network in this example was trained for 50 epochs with a batch size of 50 on an NVIDIA GeForce RTX 3090. Stochastic gradient descent optimization was used for training, and the learning rate was 10 -5 , the input face image size is 3×224×224 and the audio sequence is 1×160000.

[0122] To facilitate quantitative evaluation, this embodiment uses the accuracy index to measure performance.

[0123] Accuracy:

[0124] The quantitative comparison results are shown in Table 1

[0125] Table 1 Quantitative experiments on two datasets

[0126]

[0127] Next, conduct a qualitative evaluation:

[0128] The audio-visual matching performance is compared on the public datasets VoxCeleb and VGGFace.

[0129] The results are as follows Figure 4 As shown in the figure, adversarial methods are usually better than feature embedding-based methods in audio and video matching performance, because adversarial technology can better eliminate the heterogeneity between modalities, thereby extracting relevant cross-modal features. However, due to interference factors in the data, it is difficult to rely solely on adversarial modeling to enhance the generalization ability of the model, which also hinders the further improvement of model performance. The present invention is superior to existing adversarial methods in verification, binary matching, and multi-way matching, and significantly improves performance in both VF and FV scenarios.

[0130] In addition, this embodiment carries out 1:k multi-directional cross-modal matching experiments. As the number of matching candidates increases, the matching difficulty increases accordingly. Nevertheless, the technical solution of the present invention still shows strong competitiveness.

[0131] In addition, compared with the second-ranked method, the average performance of the present invention in VF and FV scenarios is significantly improved. This shows that the relationship-aware correction attention network can effectively bridge the connection between cross-modal features by perceiving correlation and exclude interference information. Here, VF refers to identifying multiple candidate face libraries based on audio segment recognition to achieve identity matching, such as Figure 4 (a) in ; the opposite is called FV, such as Figure 4 (b) in .

[0132] In summary, the method of the present invention can present stronger audio-visual matching capabilities on public datasets.

Claims

1. An audio-visual matching method based on relation-aware rectified attention network, characterized in that: The following steps are involved: Step 1: Get an anchor audio clip And the corresponding k face images The anchor audio is used as the recognition item and the face image is used as the gallery matching item; then the features of the anchor audio segment and the face image are extracted respectively to obtain the original audio segment features. and facial image features i represents the i-th data tuple; Step 2: The features of the audio clip and features of face images The audio clips are sent to the relation-aware rectified attention network, and the similarity between the audio clips and the single face image is calculated through the relation-aware network to obtain their respective attention matrices. and The relation-aware corrective attention network includes a parallel adaptive intra-modality corrective attention module RIRA and a relation-aware inter-modality corrective attention module AIRA; The relation-aware inter-modal correction attention module AIRA uses the adaptive attention correction unit AARU to and Adjust the attention matrix and Then, the two attention matrices are subjected to first-order inter-modal interaction and second-order inter-modal interaction in turn, and finally the feature and The adaptive intra-modality correction attention module RIRA uses the adaptive attention correction unit AARU to and Correction is performed to obtain the corrected attention matrix within the modality and Then respectively and Use decoder to extract relevant modality features and Step 3: First introduce the feature orthogonality constraint L orth , so that the features unrelated to the modality and the features related to the modality complement each other; the dynamic residual weight combination method is used to combine the dynamically selected modality internal and inter-modality enhanced features with the global features to perform the optimal matching between cross-modality features. The features of the two modalities with the best matching are recorded as and Step 4: Utilize audio features through adversarial network discriminator and facial image features To generate modality-independent features 2. The audio-visual matching method based on relation-aware corrected attention network according to claim 1, characterized in that: Step 1 also calculates the inter-modal correlation estimate pass Evaluate the semantic relevance between different modalities. The calculation formula is as follows: The above formula is obtained by conv s Convolutional layers map the semantics of audio and visual images into the same shared space; Next, Used as a pseudo-label, further supervised learning is performed by assigning its true label and calculating the relationship-aware loss L RA as follows: Among them, epoch represents the current number of iterations, N represents the total number of iterations, and σ represents the sigmoid activation function.

3. The audio-visual matching method based on relation-aware corrected attention network according to claim 1, characterized in that: The specific working content of the relation-aware inter-modal correction attention module AIRA is: First, for the similarity between the audio clip and the face image, the attention matrix is ​​obtained and The expression is as follows: In the above formula, and They refer to the attention matrices of images and audio respectively; Then, the adaptive attention correction unit AARU is used to learn the cross-modal semantic threshold to reduce the influence of interference information, and the attention matrix is ​​adjusted to obtain the adjusted attention matrix and The expression is as follows: In the above formula, ⊙ represents the product of elements, clamp represents the truncation operation, and sign represents the sign function, which is defined as follows: Next, the first-order modal interaction between audio and face image is performed, and the expression is as follows: Furthermore, the similarity matrix of the first-order inter-modal interaction is sparsely processed using the hyperparameter δ, and the semantic connection is enhanced through the intra-modal attention mechanism; finally, features are extracted through the second-order attention interaction, and the expression is as follows: In the above formula, decode s It is a decoding convolutional layer used to restore the features shared by audio and face images to the original feature dimension.

4. The audio-visual matching method based on relation-aware corrected attention network according to claim 1, characterized in that: The working content of the adaptive intra-modality correction attention module RIRA is as follows: First get the attention matrix after intra-modal correction and The expression is as follows: Among them, conv p is a dimensionality reduction convolution operation, ε is a positive threshold for online learning; for the matrix and The positive elements less than ε are set to zero by the clamp function, and the negative elements greater than -ε are set to zero by the clamp function; Then, decode them separately and get the following features: Among them, δ is a constant term, decode p It is a dimensionality-enhancing convolutional layer.

5. The audio-visual matching method based on relation-aware rectified attention network according to claim 1, characterized in that: In step 3, the feature orthogonality constraint L orth The expression is as follows: Among them, G is the maximum pooling. M represents the number of small batch data tuples. The optimal matching feature expression obtained by the dynamic residual weight combination strategy is as follows: Among them, m s represents the feature weight associated with the modality, m p represents the feature weights that are independent of the modality.

6. The audio-visual matching method based on relation-aware rectified attention network according to claim 1, characterized in that: The step 4 generates modality-independent features {h i0 ,…,h ik }, the nonlinear discrimination ability of the multilayer perceptron is used to identify potential matches, and the cross entropy method is used to calculate the loss value; Among them C m Indicates matching classification; l i is the matching identity tag; And use the relative distance stretching metric loss to expand the distance between identity-related features and narrow the interval between non-identity-related features. The cross-modal relative distance stretching metric loss is as follows: The hyperparameters μ1 and μ2 are set to 10 and 4 respectively, and the parameter θ is set to 1.2; Represents anchor samples With positive samples The Euclidean distance between is the anchor sample With negative samples The Euclidean distance between It is a positive sample With negative samples The Euclidean distance between Indicates whether the identities match, if not 1 if it matches is 0.

7. The audio-visual matching method based on relation-aware rectified attention network according to claim 1, characterized in that: The total loss is: L=L cls +L dis +L gen +L RDSM , L cls Calculate the loss for cross entropy, L RDSM Stretch metric loss for cross-modal relative distance.

Citation Information

Patent Citations

  • Audio-visual video analysis device and method based on multi-scale semantic network

    CN114519809A

  • Audio-visual emotion recognition method and system based on multi-scale and global cross attention

    CN116758451A

  • Double-flow network cross-modal mouth shape synchronization method and system based on channel attention

    CN117176998A

  • Systems, Methods and Computer Program Products for Associating Media Content Having Different Modalities

    US20200394213A1

  • Audio-visual assisted fine-grained tactile signal reconstruction method

    WO2024104376A1