An audio-video fusion method, device, equipment and medium
By extracting and splitting features from audio and video segments and using an attention weight matrix to determine the temporal alignment relationship, the problem of audio and video asynchrony in audio-video fusion is solved, achieving high-precision audio-video temporal alignment and audio-visual synchronization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANDONG INSPUR SCI RES INST CO LTD
- Filing Date
- 2025-09-01
- Publication Date
- 2026-07-21
Smart Images
Figure CN121037600B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multimedia data processing, and in particular to an audio-visual fusion method, apparatus, device, and medium. Background Technology
[0002] Audio-video fusion technology differs from simple audio and video splicing. It requires the fusion of audio and video data to keep the sound and picture synchronized. Current audio-video fusion mainly relies on simple timestamp alignment and directly merging audio and video data corresponding to the same timestamp. However, due to the significant differences in the representation of audio and video data, as well as factors such as network latency, encoding / decoding processing, and hardware performance differences, the accuracy of timestamp alignment may decrease, resulting in audio and video desynchronization in the fused audio-video.
[0003] Therefore, how to improve the accuracy of audio and video timing alignment and ensure that the fused audio and video remain synchronized is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention
[0004] In view of this, the purpose of this invention is to provide an audio-video fusion method, apparatus, device, and medium that can improve the accuracy of audio-video timing alignment and ensure that the fused audio and video maintain audio-visual synchronization. The specific solution is as follows:
[0005] Firstly, this application provides an audio-visual fusion method, including:
[0006] Feature extraction is performed on audio and video segments to obtain initial audio and video feature sequences with the same sequence length and feature dimension;
[0007] Based on the feature dimension, each initial feature sequence is split into a common feature sequence and a private feature sequence with constant sequence length;
[0008] Based on the attention weight matrix between each feature vector in the audio common feature sequence and the video common feature sequence, the temporal alignment relationship between the audio common feature sequence and the video common feature sequence is determined;
[0009] The common audio feature sequence and the common video feature sequence are aligned using the temporal alignment relationship, and the private audio feature sequence and the private video feature sequence are respectively concatenated to the aligned common audio feature sequence and the common video feature sequence to obtain the aligned audio reconstruction feature sequence and video reconstruction feature sequence;
[0010] The audio reconstruction feature sequence and the video reconstruction feature sequence are fused to obtain a fused audio and video segment.
[0011] Optionally, the step of splitting each initial feature sequence into a common feature sequence and a private feature sequence with constant sequence length based on the feature dimension includes:
[0012] From each feature vector contained in each initial feature sequence, obtain sub-vectors of the same feature dimension;
[0013] Based on each sub-vector of any feature dimension in each initial feature sequence and each sub-vector of other feature dimensions, determine the correlation between any feature dimension and the other feature dimensions;
[0014] The cross-dimensional correlation of any feature dimension is determined based on the mean correlation between any feature dimension and the other feature dimensions.
[0015] Based on the order of cross-dimensional relevance from high to low, each feature dimension is divided into a first feature dimension and a second feature dimension.
[0016] Based on the first feature dimension and the second feature dimension, each initial feature sequence is split into a common feature sequence and a private feature sequence with unchanged sequence length.
[0017] Optionally, determining the temporal alignment relationship between the audio common feature sequence and the video common feature sequence based on the attention weight matrix between each feature vector in the audio common feature sequence and the video common feature sequence includes:
[0018] Based on the multi-head attention mechanism, the attention weight matrix between each feature vector in the audio common feature sequence and the video common feature sequence is determined;
[0019] Based on the maximum weight of each row or column in the attention weight matrix, determine the correspondence between each feature vector in the audio common feature sequence and each feature vector in the video common feature sequence;
[0020] Based on the correspondence, the temporal alignment relationship between the audio common feature sequence and the video common feature sequence is determined.
[0021] Optionally, the step of determining the attention weight matrix between each feature vector in the audio common feature sequence and the video common feature sequence based on the multi-head attention mechanism includes:
[0022] Based on each attention head, a sub-weight matrix is determined between each feature vector in the audio common feature sequence and the video common feature sequence; each attention head includes a first attention head, a second attention head, and a third attention head;
[0023] Using the preset weights corresponding to each attention head, the sub-weight matrices corresponding to each attention head are weighted and fused to obtain the attention weight matrix.
[0024] Optionally, based on the first attention head, a sub-weight matrix is determined between each feature vector in the audio common feature sequence and the video common feature sequence, including:
[0025] The common audio feature sequence and the common video feature sequence are input into the first attention head, so that the first attention head directly determines the first similarity between the first feature vector in the common audio feature sequence and each feature vector in the common video feature sequence based on the first feature vector in the common audio feature sequence and each feature vector in the common video feature sequence, and calculates the sum of the first similarities; the first feature vector is any feature vector in the common audio feature sequence;
[0026] A first attention weight between the first feature vector and the second feature vector in the video common feature sequence is determined based on the ratio between the first similarity between the first feature vector and the second feature vector in the video common feature sequence and the sum of the first similarities; the second feature vector is any feature vector in the video common feature sequence.
[0027] Based on the first attention weights between each feature vector in the audio common feature sequence and each feature vector in the video common feature sequence, a sub-weight matrix is determined between each feature vector in the audio common feature sequence and each feature vector in the video common feature sequence.
[0028] Optionally, based on the second attention head, a sub-weight matrix is determined between each feature vector in the audio common feature sequence and the video common feature sequence, including:
[0029] The common audio feature sequence and the common video feature sequence are input into the second attention head, so that the second attention head performs difference operations on each feature vector in the common audio feature sequence and the common video feature sequence to obtain the corresponding difference results;
[0030] Based on the difference results corresponding to the first feature vector and each feature vector in the video common feature sequence, the second similarity between the first feature vector and each feature vector in the video common feature sequence is determined, and the sum of the second similarities is calculated.
[0031] A second attention weight between the first feature vector and the second feature vector is determined based on the ratio between the second similarity between the first feature vector and the second feature vector and the sum of the second similarities.
[0032] Based on the second attention weights between each feature vector in the audio common feature sequence and each feature vector in the video common feature sequence, a sub-weight matrix is determined between each feature vector in the audio common feature sequence and each feature vector in the video common feature sequence.
[0033] Optionally, based on the third attention head, a sub-weight matrix is determined between each feature vector in the audio common feature sequence and the video common feature sequence, including:
[0034] The common audio feature sequence and the common video feature sequence are input to the third attention head, so that the third attention head performs a second norm operation on each feature vector in the common audio feature sequence and the common video feature sequence to obtain the corresponding norm operation result;
[0035] Based on the difference between the norm operation result corresponding to the first feature vector and the norm operation result corresponding to each feature vector in the video common feature sequence, the third similarity between the first feature vector and each feature vector in the video common feature sequence is determined, and the sum of the third similarities is calculated.
[0036] The third attention weight between the first feature vector and the second feature vector is determined based on the ratio between the third similarity between the first feature vector and the second feature vector and the sum of the third similarities.
[0037] Based on the third attention weights between each feature vector in the audio common feature sequence and each feature vector in the video common feature sequence, a sub-weight matrix is determined between each feature vector in the audio common feature sequence and each feature vector in the video common feature sequence.
[0038] Secondly, this application provides an audio-visual fusion device, comprising:
[0039] The feature extraction module is used to extract features from audio and video segments to obtain initial audio and video feature sequences with the same sequence length and feature dimensions.
[0040] The feature sequence splitting module is used to split each initial feature sequence into a common feature sequence and a private feature sequence with constant sequence length based on the feature dimension.
[0041] The alignment relationship determination module is used to determine the temporal alignment relationship between the audio common feature sequence and the video common feature sequence based on the attention weight matrix between each feature vector in the audio common feature sequence and the video common feature sequence;
[0042] The temporal alignment module is used to align the audio common feature sequence and the video common feature sequence using the temporal alignment relationship, and to concatenate the audio private feature sequence and the video private feature sequence to the aligned audio common feature sequence and the video common feature sequence, respectively, to obtain the aligned audio reconstruction feature sequence and video reconstruction feature sequence;
[0043] The audio-video fusion module is used to fuse the audio reconstruction feature sequence and the video reconstruction feature sequence to obtain a fused audio-video segment.
[0044] Thirdly, this application provides an electronic device, comprising:
[0045] Memory, used to store computer programs;
[0046] A processor for executing the computer program to implement the aforementioned audio-video fusion method.
[0047] Fourthly, this application provides a computer-readable storage medium for storing a computer program that, when executed by a processor, implements the aforementioned audio-video fusion method.
[0048] In this application, feature extraction is performed on audio and video segments to obtain initial audio and video feature sequences with the same sequence length and feature dimension. Each initial feature sequence is then split into a common feature sequence and a private feature sequence with constant sequence length based on the feature dimension. The temporal alignment relationship between the audio and video common feature sequences is determined based on the attention weight matrix between each feature vector in the audio and video common feature sequences. The audio and video common feature sequences are aligned using this temporal alignment relationship, and the audio and video private feature sequences are concatenated to the aligned audio and video common feature sequences to obtain aligned audio and video reconstructed feature sequences. Finally, the audio and video reconstructed feature sequences are fused to obtain a fused audio-video segment.
[0049] Therefore, this application splits both the initial audio and video feature sequences into common feature sequences and private feature sequences with constant sequence lengths based on feature dimensions. Considering that the private feature sequences retain their unique feature information, if the temporal alignment relationship between audio and video is determined based on the attention weight matrix between the audio and video private feature sequences, the temporal alignment accuracy may decrease due to the unique properties of the private features. Therefore, this application determines the temporal alignment relationship between audio and video only based on the attention weight matrix between the audio and video common feature sequences. After aligning the audio and video common feature sequences using the temporal alignment relationship, the audio and video private feature sequences are respectively concatenated to the aligned audio and video common feature sequences. This not only ensures the accuracy of audio-video temporal alignment but also preserves the private features of audio and video. On this basis, audio-video fusion can not only generate fused audio and video containing rich feature information but also ensure that the fused audio and video maintain audio-visual synchronization. Attached Figure Description
[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0051] Figure 1 This is a flowchart of an audio-visual fusion method disclosed in this application;
[0052] Figure 2 This is a schematic diagram of the structure of an audio-visual fusion device disclosed in this application;
[0053] Figure 3 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation
[0054] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0055] Existing audio-video fusion methods primarily rely on simple timestamp alignment and directly merging audio and video data corresponding to the same timestamp. However, due to the significant differences in the representation of audio and video data, as well as factors such as network latency, encoding / decoding processing, and hardware performance differences, the accuracy of timestamp alignment may decrease, leading to audio and video desynchronization in the fused audio-video. Therefore, this application provides an audio-video fusion method that can improve the accuracy of audio-video timing alignment and ensure that the fused audio-video maintains audio-visual synchronization.
[0056] See Figure 1 As shown, this embodiment of the invention discloses an audio-visual fusion method, including:
[0057] Step S11: Extract features from the audio and video segments to obtain initial audio and video feature sequences with the same sequence length and feature dimensions.
[0058] In this embodiment of the invention, audio segments and video segments of the same duration are obtained, and feature extraction is performed on the audio segments to obtain an initial audio feature sequence, and feature extraction is performed on the video segments to obtain an initial video feature sequence.
[0059] The initial audio feature sequence and the initial video feature sequence have the same sequence length, meaning they contain the same number of feature vectors. Furthermore, the initial audio feature sequence and the initial video feature sequence have the same feature dimension, meaning each feature vector in both sequences has the same feature dimension, or in other words, each feature vector in both sequences contains the same number of sub-vectors.
[0060] In one example, a short-time Fourier transform (STFT) is performed on the audio segment to convert it from the time domain to the frequency domain, resulting in an initial audio feature sequence of length T and feature dimension 512. Similarly, a deep convolutional neural network (DCNN) is used to extract features from the video segment, yielding an initial video feature sequence of length T and feature dimension 512.
[0061] Step S12: Based on the feature dimension, split each initial feature sequence into a common feature sequence and a private feature sequence with the same sequence length.
[0062] In this embodiment of the invention, after obtaining the initial audio feature sequence and the initial video feature sequence, each initial feature sequence is split into a common feature sequence and a private feature sequence with constant sequence length based on the feature dimension. It should be noted that the sequence lengths of both the common feature sequence and the private feature sequence are consistent with those of each initial feature sequence, and the feature dimensions of the common feature sequence and the private feature sequence together constitute the feature dimension of each initial feature sequence.
[0063] Specifically, in this embodiment of the invention, the initial audio feature sequence is split into a common audio feature sequence and a private audio feature sequence with constant sequence length based on the feature dimension, and the initial video feature sequence is split into a common video feature sequence and a private video feature sequence with constant sequence length based on the feature dimension.
[0064] Specifically, from the feature vectors contained in each initial feature sequence, sub-vectors of the same feature dimension are obtained; based on the sub-vectors of each feature dimension in each initial feature sequence and the sub-vectors of other feature dimensions, the correlation between any feature dimension and other feature dimensions is determined; based on the mean of the correlation between any feature dimension and other feature dimensions, the cross-dimensional correlation of any feature dimension is determined; according to the order of cross-dimensional correlation from high to low, each feature dimension is equally divided into a first feature dimension and a second feature dimension; based on the first feature dimension and the second feature dimension, each initial feature sequence is split into a common feature sequence and a private feature sequence with unchanged sequence length.
[0065] Taking a sequence length of T and a feature dimension of 512 as an example, from the T feature vectors contained in each initial feature sequence, the sub-vectors of the 1st feature dimension, the 2nd feature dimension, ..., the 512th feature dimension are obtained respectively. Based on the sub-vectors of the i-th feature dimension and the sub-vectors of other feature dimensions, the correlation between the i-th feature dimension and other feature dimensions is determined. The average of the correlation between the i-th feature dimension and other feature dimensions is calculated to obtain the cross-dimensional correlation of the i-th feature dimension. Then, according to the order of cross-dimensional correlation from high to low, the 512 feature dimensions are divided into the first feature dimension and the second feature dimension. The first feature dimension and the second feature dimension are both 256, and the cross-dimensional correlation of the first feature dimension is higher than that of the second feature dimension. Finally, based on the first and second feature dimensions, each initial feature sequence is split into a common feature sequence and a private feature sequence with the same sequence length. That is, the common feature sequence is a sequence constructed based on the sub-vectors of the first feature dimension in each initial feature sequence, and the private feature sequence is a sequence constructed based on the sub-vectors of the second feature dimension in each initial feature sequence.
[0066] Step S13: Based on the attention weight matrix between each feature vector in the audio common feature sequence and the video common feature sequence, determine the temporal alignment relationship between the audio common feature sequence and the video common feature sequence.
[0067] In this embodiment of the invention, based on a multi-head attention mechanism, an attention weight matrix is determined between each feature vector in the audio common feature sequence and the video common feature sequence; according to the maximum weight of each row or column in the attention weight matrix, the correspondence between each feature vector in the audio common feature sequence and the feature vector in the video common feature sequence is determined; based on the correspondence, the temporal alignment relationship between the audio common feature sequence and the video common feature sequence is determined.
[0068] Specifically, for a multi-head attention mechanism, it may include: determining sub-weight matrices between feature vectors in the audio common feature sequence and the video common feature sequence based on each attention head; wherein each attention head includes a first attention head, a second attention head, and a third attention head; and using the preset weights corresponding to each attention head, weighting and fusing the sub-weight matrices corresponding to each attention head to obtain an attention weight matrix.
[0069] For the first attention head, the common audio feature sequence and the common video feature sequence are input into the first attention head so that the first attention head can directly determine the first similarity between the first feature vector in the common audio feature sequence and each feature vector in the common video feature sequence based on the first feature vector in the common audio feature sequence and each feature vector in the common video feature sequence, and calculate the sum of the first similarities; wherein the first feature vector is any feature vector in the common audio feature sequence; the first attention weight between the first feature vector and the second feature vector in the common video feature sequence is determined according to the ratio between the first similarity between the first feature vector and the second feature vector in the common video feature sequence and the sum of the first similarities; wherein the second feature vector is any feature vector in the common video feature sequence; and the sub-weight matrix between each feature vector in the common audio feature sequence and each feature vector in the common video feature sequence is determined based on the first attention weight between each feature vector in the common audio feature sequence and each feature vector in the common video feature sequence.
[0070] It should be noted that the formula for calculating the first attention weight is as follows:
[0071] ;
[0072] in, This represents the i-th feature vector in the sequence of common audio features; represents the j-th feature vector in the common feature sequence of the video; d represents the feature dimension of the feature vector. Let represent the k-th feature vector in the common feature sequence of the video, where k = 1, 2, ..., T if the sequence length is T; exp represents the exponential function. This represents the first similarity between the i-th feature vector in the audio common feature sequence and the j-th feature vector in the video common feature sequence; This represents the first similarity between the i-th feature vector in the audio common feature sequence and the k-th feature vector in the video common feature sequence; This represents the sum of the first similarity scores; This represents the first attention weight between the i-th feature vector in the audio common feature sequence and the j-th feature vector in the video common feature sequence.
[0073] For the second attention head, the common audio feature sequence and the common video feature sequence are input into the second attention head so that the second attention head performs difference operations on each feature vector in the common audio feature sequence and the common video feature sequence to obtain the corresponding difference results. Based on the difference results corresponding to each feature vector in the first feature vector and each feature vector in the common video feature sequence, the second similarity between the first feature vector and each feature vector in the common video feature sequence is determined, and the sum of the second similarities is calculated. According to the ratio between the second similarity between the first feature vector and the second feature vector and the sum of the second similarities, the second attention weight between the first feature vector and the second feature vector is determined. Based on the second attention weight between each feature vector in the common audio feature sequence and each feature vector in the common video feature sequence, the sub-weight matrix between each feature vector in the common audio feature sequence and the common video feature sequence is determined.
[0074] Specifically, the first feature vector in the audio common feature sequence and the video common feature sequence can be obtained by forward differencing; the last feature vector in the audio common feature sequence and the video common feature sequence can be obtained by backward differencing; and the other feature vectors in the audio common feature sequence and the video common feature sequence can be obtained by central differencing.
[0075] It should be noted that the formula for calculating the second attention weight is as follows:
[0076] ;
[0077] in, This represents the difference result corresponding to the i-th feature vector in the common feature sequence of audio; d represents the difference result corresponding to the j-th feature vector in the common feature sequence of the video; d represents the feature dimension of the feature vector. This represents the difference result corresponding to the k-th feature vector in the common feature sequence of the video, where if the sequence length is T, then k = 1, 2, ..., T; exp represents the exponential function. This represents the second similarity between the i-th feature vector in the audio common feature sequence and the j-th feature vector in the video common feature sequence; This represents the second similarity between the i-th feature vector in the audio common feature sequence and the k-th feature vector in the video common feature sequence; This represents the sum of the second similarities; This represents the second attention weight between the i-th feature vector in the audio common feature sequence and the j-th feature vector in the video common feature sequence.
[0078] For the third attention head, the common audio feature sequence and the common video feature sequence are input into the third attention head so that the third attention head performs L2 norm operations on each feature vector in the common audio feature sequence and the common video feature sequence to obtain the corresponding norm operation results. Based on the difference between the norm operation results corresponding to the first feature vector and the norm operation results corresponding to each feature vector in the common video feature sequence, the third similarity between the first feature vector and each feature vector in the common video feature sequence is determined, and the sum of the third similarities is calculated. According to the ratio between the third similarity between the first feature vector and the second feature vector and the sum of the third similarities, the third attention weight between the first feature vector and the second feature vector is determined. Based on the third attention weight between each feature vector in the common audio feature sequence and each feature vector in the common video feature sequence, the sub-weight matrix between each feature vector in the common audio feature sequence and the common video feature sequence is determined.
[0079] Among them, for each feature vector in the common feature sequence of audio, the following is adopted: Perform L2 norm operations. This represents the t-th feature vector in the sequence of common audio features. This represents the norm operation result corresponding to the t-th eigenvector in the common feature sequence of audio.
[0080] Specifically, for the first feature vector in the common feature sequence of the video, the following is adopted: Perform L2 norm operations. This represents the first feature vector in the sequence of common features in the video. This represents the norm operation result corresponding to the first feature vector in the video common feature sequence. For other feature vectors in the video common feature sequence, the norm operation result is... Perform L2 norm operations. This represents the t-th feature vector in the common feature sequence of the video. This represents the (t-1)th feature vector in the common feature sequence of the video. This represents the norm operation result corresponding to the t-th feature vector in the common feature sequence of the video.
[0081] It should be noted that the formula for calculating the third attention weight is as follows:
[0082] ;
[0083] in, This represents the norm operation result corresponding to the i-th eigenvector in the common audio feature sequence; This represents the norm operation result corresponding to the j-th feature vector in the common feature sequence of the video; Indicates the preset temperature coefficient; This represents the norm operation result corresponding to the k-th feature vector in the common feature sequence of the video, where k = 1, 2, ..., T if the sequence length is T; exp represents the exponential function. The third similarity is represented between the i-th feature vector in the audio common feature sequence and the j-th feature vector in the video common feature sequence; The third similarity is represented between the i-th feature vector in the audio common feature sequence and the k-th feature vector in the video common feature sequence; This represents the sum of the third similarity; This represents the third attention weight between the i-th feature vector in the audio common feature sequence and the j-th feature vector in the video common feature sequence.
[0084] Step S14: Align the audio common feature sequence and the video common feature sequence using the temporal alignment relationship, and then concatenate the audio private feature sequence and the video private feature sequence to the aligned audio common feature sequence and the video common feature sequence, respectively, to obtain the aligned audio reconstruction feature sequence and video reconstruction feature sequence.
[0085] In this embodiment of the invention, after determining the temporal alignment relationship between the audio common feature sequence and the video common feature sequence, the audio common feature sequence and the video common feature sequence are temporally aligned using the temporal alignment relationship to obtain aligned audio common feature sequences and video common feature sequences. Then, the audio private feature sequence and the video private feature sequence are respectively concatenated to the aligned audio common feature sequence and video common feature sequence to obtain temporally aligned audio reconstruction feature sequence and video reconstruction feature sequence. At this time, the sequence length and feature dimension of the audio reconstruction feature sequence are the same as those of the initial audio feature sequence, and the sequence length and feature dimension of the video reconstruction feature sequence are the same as those of the initial video feature sequence.
[0086] Step S15: Fuse the audio reconstruction feature sequence and the video reconstruction feature sequence to obtain a fused audio and video segment.
[0087] In this embodiment of the invention, feature fusion is performed on the time-aligned audio reconstruction feature sequence and video reconstruction feature sequence to fuse audio reconstruction features and video reconstruction features corresponding to the same time sequence, thereby obtaining an audio-video fusion feature sequence. Finally, the fused audio and video segments can be obtained based on the audio-video fusion feature sequence. The audio-video fusion feature sequence has the same sequence length as both the audio reconstruction feature sequence and the video reconstruction feature sequence.
[0088] Therefore, this application splits both the initial audio and video feature sequences into common feature sequences and private feature sequences with constant sequence lengths based on feature dimensions. Considering that the private feature sequences retain their unique feature information, if the temporal alignment relationship between audio and video is determined based on the attention weight matrix between the audio and video private feature sequences, the temporal alignment accuracy may decrease due to the unique properties of the private features. Therefore, this application determines the temporal alignment relationship between audio and video only based on the attention weight matrix between the audio and video common feature sequences. After aligning the audio and video common feature sequences using the temporal alignment relationship, the audio and video private feature sequences are respectively concatenated to the aligned audio and video common feature sequences. This not only ensures the accuracy of audio-video temporal alignment but also preserves the private features of audio and video. On this basis, audio-video fusion can not only generate fused audio and video containing rich feature information but also ensure that the fused audio and video maintain audio-visual synchronization.
[0089] See Figure 2 As shown, an embodiment of the present invention discloses an audio-visual fusion device, comprising:
[0090] The feature extraction module 11 is used to extract features from audio and video segments to obtain initial audio feature sequences and initial video feature sequences with the same sequence length and feature dimensions.
[0091] The feature sequence splitting module 12 is used to split each initial feature sequence into a common feature sequence and a private feature sequence with constant sequence length based on the feature dimension.
[0092] Alignment determination module 13 is used to determine the temporal alignment relationship between the audio common feature sequence and the video common feature sequence based on the attention weight matrix between each feature vector in the audio common feature sequence and the video common feature sequence;
[0093] The temporal alignment module 14 is used to align the audio common feature sequence and the video common feature sequence using the temporal alignment relationship, and to concatenate the audio private feature sequence and the video private feature sequence to the aligned audio common feature sequence and the video common feature sequence, respectively, to obtain the aligned audio reconstruction feature sequence and video reconstruction feature sequence;
[0094] The audio-video fusion module 15 is used to fuse the audio reconstruction feature sequence and the video reconstruction feature sequence to obtain the fused audio-video segment.
[0095] Therefore, this application splits both the initial audio and video feature sequences into common feature sequences and private feature sequences with constant sequence lengths based on feature dimensions. Considering that the private feature sequences retain their unique feature information, if the temporal alignment relationship between audio and video is determined based on the attention weight matrix between the audio and video private feature sequences, the temporal alignment accuracy may decrease due to the unique properties of the private features. Therefore, this application determines the temporal alignment relationship between audio and video only based on the attention weight matrix between the audio and video common feature sequences. After aligning the audio and video common feature sequences using the temporal alignment relationship, the audio and video private feature sequences are respectively concatenated to the aligned audio and video common feature sequences. This not only ensures the accuracy of audio-video temporal alignment but also preserves the private features of audio and video. On this basis, audio-video fusion can not only generate fused audio and video containing rich feature information but also ensure that the fused audio and video maintain audio-visual synchronization.
[0096] In some specific embodiments, the feature sequence splitting module 12 includes:
[0097] The sub-vector acquisition unit is used to acquire sub-vectors of the same feature dimension from the feature vectors contained in each initial feature sequence;
[0098] The correlation determination unit is used to determine the correlation between any feature dimension and the other feature dimensions based on each sub-vector of any feature dimension in each initial feature sequence and each sub-vector of other feature dimensions.
[0099] The correlation degree determination unit is used to determine the cross-dimensional correlation degree of any feature dimension based on the mean of the correlation degree between any feature dimension and the other feature dimensions;
[0100] The feature dimension partitioning unit is used to divide each feature dimension into a first feature dimension and a second feature dimension in descending order of cross-dimensional correlation.
[0101] The feature sequence splitting unit is used to split each initial feature sequence into a common feature sequence and a private feature sequence with unchanged sequence length based on the first feature dimension and the second feature dimension.
[0102] In some specific embodiments, the alignment relationship determination module 13 includes:
[0103] The matrix determination submodule is used to determine the attention weight matrix between each feature vector in the audio common feature sequence and the video common feature sequence based on the multi-head attention mechanism;
[0104] The correspondence determination unit is used to determine the correspondence between each feature vector in the audio common feature sequence and each feature vector in the video common feature sequence based on the maximum weight of each row or column in the attention weight matrix.
[0105] The alignment relationship determination unit is used to determine the temporal alignment relationship between the audio common feature sequence and the video common feature sequence based on the correspondence relationship.
[0106] In some specific embodiments, the matrix determination submodule includes:
[0107] The submatrix determination unit is used to determine the subweight matrix between each feature vector in the audio common feature sequence and the video common feature sequence based on each attention head; the attention head includes a first attention head, a second attention head, and a third attention head;
[0108] The weight matrix determination unit is used to perform weighted fusion of the sub-weight matrices corresponding to each attention head using the preset weights corresponding to each attention head, so as to obtain the attention weight matrix.
[0109] In some specific embodiments, the submatrix determination unit includes:
[0110] The first similarity calculation unit is used to input the audio common feature sequence and the video common feature sequence into the first attention head, so that the first attention head directly determines the first similarity between the first feature vector and each feature vector in the video common feature sequence based on the first feature vector in the audio common feature sequence and each feature vector in the video common feature sequence, and calculates the sum of the first similarities; the first feature vector is any feature vector in the audio common feature sequence;
[0111] The first weight determination unit is used to determine a first attention weight between the first feature vector and the second feature vector based on the ratio between the first similarity between the first feature vector and the second feature vector in the video common feature sequence and the sum of the first similarities; the second feature vector is any feature vector in the video common feature sequence.
[0112] The first matrix determination unit is used to determine the sub-weight matrix between each feature vector in the audio common feature sequence and each feature vector in the video common feature sequence based on the first attention weight between each feature vector in the audio common feature sequence and each feature vector in the video common feature sequence.
[0113] In some specific embodiments, the submatrix determination unit includes:
[0114] The vector difference unit is used to input the audio common feature sequence and the video common feature sequence into the second attention head, so that the second attention head performs difference operations on each feature vector in the audio common feature sequence and the video common feature sequence to obtain the corresponding difference results;
[0115] The second similarity calculation unit is used to determine the second similarity between the first feature vector and each feature vector in the video common feature sequence based on the difference results corresponding to the first feature vector and each feature vector in the video common feature sequence, and to calculate the sum of the second similarities;
[0116] The second weight determination unit is used to determine the second attention weight between the first feature vector and the second feature vector based on the ratio between the second similarity between the first feature vector and the second feature vector and the sum of the second similarities.
[0117] The second matrix determination unit is used to determine the sub-weight matrix between each feature vector in the audio common feature sequence and each feature vector in the video common feature sequence based on the second attention weight between each feature vector in the audio common feature sequence and each feature vector in the video common feature sequence.
[0118] In some specific embodiments, the submatrix determination unit includes:
[0119] The norm operation unit is used to input the audio common feature sequence and the video common feature sequence into the third attention head, so that the third attention head performs a second norm operation on each feature vector in the audio common feature sequence and the video common feature sequence to obtain the corresponding norm operation result;
[0120] The third similarity calculation unit is used to determine the third similarity between the first feature vector and each feature vector in the video common feature sequence based on the difference between the norm operation result corresponding to the first feature vector and the norm operation result corresponding to each feature vector in the video common feature sequence, and to calculate the sum of the third similarities.
[0121] The third weight determination unit is used to determine the third attention weight between the first feature vector and the second feature vector based on the ratio between the third similarity between the first feature vector and the second feature vector and the sum of the third similarities.
[0122] The third matrix determination unit is used to determine the sub-weight matrix between each feature vector in the audio common feature sequence and each feature vector in the video common feature sequence based on the third attention weight between each feature vector in the audio common feature sequence and each feature vector in the video common feature sequence.
[0123] Furthermore, embodiments of this application also disclose an electronic device, Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.
[0124] Figure 3 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the audio-visual fusion method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.
[0125] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.
[0126] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.
[0127] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the audio-visual fusion method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.
[0128] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned audio-video fusion method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.
[0129] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.
[0130] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0131] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0132] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0133] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. An audio-video fusion method, characterized in that, include: Feature extraction is performed on audio and video segments to obtain initial audio and video feature sequences with the same sequence length and feature dimension; Based on the feature dimension, each initial feature sequence is split into a common feature sequence and a private feature sequence with constant sequence length; Based on the attention weight matrix between each feature vector in the audio common feature sequence and the video common feature sequence, the temporal alignment relationship between the audio common feature sequence and the video common feature sequence is determined; The common audio feature sequence and the common video feature sequence are aligned using the temporal alignment relationship, and the private audio feature sequence and the private video feature sequence are respectively concatenated to the aligned common audio feature sequence and the common video feature sequence to obtain the aligned audio reconstruction feature sequence and video reconstruction feature sequence; The audio reconstruction feature sequence and the video reconstruction feature sequence are fused to obtain a fused audio and video segment; Specifically, the step of splitting each initial feature sequence into a common feature sequence and a private feature sequence with constant sequence length based on feature dimension includes: From each feature vector contained in each initial feature sequence, obtain sub-vectors of the same feature dimension; Based on each sub-vector of any feature dimension in each initial feature sequence and each sub-vector of other feature dimensions, determine the correlation between any feature dimension and the other feature dimensions; The cross-dimensional correlation of any feature dimension is determined based on the mean correlation between any feature dimension and the other feature dimensions. Based on the order of cross-dimensional relevance from high to low, each feature dimension is divided into a first feature dimension and a second feature dimension. Based on the first feature dimension and the second feature dimension, each initial feature sequence is split into a common feature sequence and a private feature sequence with constant sequence length; The step of determining the temporal alignment relationship between the audio common feature sequence and the video common feature sequence based on the attention weight matrix between each feature vector in the audio common feature sequence and the video common feature sequence includes: Based on the multi-head attention mechanism, the attention weight matrix between each feature vector in the audio common feature sequence and the video common feature sequence is determined; Based on the maximum weight of each row or column in the attention weight matrix, determine the correspondence between each feature vector in the audio common feature sequence and each feature vector in the video common feature sequence; Based on the correspondence, the temporal alignment relationship between the audio common feature sequence and the video common feature sequence is determined.
2. The audio-visual fusion method according to claim 1, characterized in that, The multi-head attention mechanism is used to determine the attention weight matrix between each feature vector in the audio common feature sequence and the video common feature sequence, including: Based on each attention head, a sub-weight matrix is determined between each feature vector in the audio common feature sequence and the video common feature sequence; each attention head includes a first attention head, a second attention head, and a third attention head; Using the preset weights corresponding to each attention head, the sub-weight matrices corresponding to each attention head are weighted and fused to obtain the attention weight matrix.
3. The audio-visual fusion method according to claim 2, characterized in that, Based on the first attention head, a sub-weight matrix is determined between each feature vector in the audio common feature sequence and the video common feature sequence, including: The common audio feature sequence and the common video feature sequence are input into the first attention head, so that the first attention head directly determines the first similarity between the first feature vector in the common audio feature sequence and each feature vector in the common video feature sequence based on the first feature vector in the common audio feature sequence and each feature vector in the common video feature sequence, and calculates the sum of the first similarities; the first feature vector is any feature vector in the common audio feature sequence; A first attention weight between the first feature vector and the second feature vector in the video common feature sequence is determined based on the ratio between the first similarity between the first feature vector and the second feature vector in the video common feature sequence and the sum of the first similarities; the second feature vector is any feature vector in the video common feature sequence. Based on the first attention weights between each feature vector in the audio common feature sequence and each feature vector in the video common feature sequence, a sub-weight matrix is determined between each feature vector in the audio common feature sequence and each feature vector in the video common feature sequence.
4. The audio-visual fusion method according to claim 3, characterized in that, Based on the second attention head, a sub-weight matrix is determined between each feature vector in the audio common feature sequence and the video common feature sequence, including: The common audio feature sequence and the common video feature sequence are input into the second attention head, so that the second attention head performs difference operations on each feature vector in the common audio feature sequence and the common video feature sequence to obtain the corresponding difference results; Based on the difference results corresponding to the first feature vector and each feature vector in the video common feature sequence, the second similarity between the first feature vector and each feature vector in the video common feature sequence is determined, and the sum of the second similarities is calculated. A second attention weight between the first feature vector and the second feature vector is determined based on the ratio between the second similarity between the first feature vector and the second feature vector and the sum of the second similarities. Based on the second attention weights between each feature vector in the audio common feature sequence and each feature vector in the video common feature sequence, a sub-weight matrix is determined between each feature vector in the audio common feature sequence and each feature vector in the video common feature sequence.
5. The audio-visual fusion method according to claim 3, characterized in that, Based on the third attention head, the sub-weight matrix between each feature vector in the audio common feature sequence and the video common feature sequence is determined, including: The common audio feature sequence and the common video feature sequence are input to the third attention head, so that the third attention head performs a second norm operation on each feature vector in the common audio feature sequence and the common video feature sequence to obtain the corresponding norm operation result; Based on the difference between the norm operation result corresponding to the first feature vector and the norm operation result corresponding to each feature vector in the video common feature sequence, the third similarity between the first feature vector and each feature vector in the video common feature sequence is determined, and the sum of the third similarities is calculated. The third attention weight between the first feature vector and the second feature vector is determined based on the ratio between the third similarity between the first feature vector and the second feature vector and the sum of the third similarities. Based on the third attention weights between each feature vector in the audio common feature sequence and each feature vector in the video common feature sequence, a sub-weight matrix is determined between each feature vector in the audio common feature sequence and each feature vector in the video common feature sequence.
6. An audio-visual fusion device, characterized in that, The audio-video fusion device is used to implement the audio-video fusion method as described in any one of claims 1 to 5, and the audio-video fusion device includes: The feature extraction module is used to extract features from audio and video segments to obtain initial audio and video feature sequences with the same sequence length and feature dimensions. The feature sequence splitting module is used to split each initial feature sequence into a common feature sequence and a private feature sequence with constant sequence length based on the feature dimension. The alignment relationship determination module is used to determine the temporal alignment relationship between the audio common feature sequence and the video common feature sequence based on the attention weight matrix between each feature vector in the audio common feature sequence and the video common feature sequence; The temporal alignment module is used to align the audio common feature sequence and the video common feature sequence using the temporal alignment relationship, and to concatenate the audio private feature sequence and the video private feature sequence to the aligned audio common feature sequence and the video common feature sequence, respectively, to obtain the aligned audio reconstruction feature sequence and video reconstruction feature sequence; The audio-video fusion module is used to fuse the audio reconstruction feature sequence and the video reconstruction feature sequence to obtain a fused audio-video segment.
7. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the audio-video fusion method as described in any one of claims 1 to 5.
8. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the audio-video fusion method as described in any one of claims 1 to 5.