A weakly supervised speech-video localization method and system based on semantic interaction

Abstract

This invention discloses a weakly supervised speech-video localization method and system based on semantic interaction, belonging to the field of visual-language cross-modal localization. First, this invention uses a pre-trained dataset consisting of speech and corresponding text to pre-train a speech encoder based on four collaborative tasks. Then, it uses a localization task dataset consisting of speech and corresponding video to construct positive and negative sample pairs. Based on the prediction results of the localization network, a binary cross-entropy loss function is calculated. Furthermore, using the candidate segments with the highest scores of the positive sample pairs predicted by the localization network, frame-level positive and negative samples are constructed. Through acoustic-visual contrastive learning, a loss function based on maximizing mutual information is obtained. Combining these two losses, the localization network is weakly trained for localization, achieving efficient cross-modal interaction and improving the accuracy of speech-video localization.

Description

Technical Field

[0001] This invention relates to the field of weakly supervised speech-video localization, and more particularly to a weakly supervised speech-video localization method and system based on semantic interaction. Background Technology

[0002] Currently, query-based video localization has attracted widespread attention and become an important artificial intelligence task. This task aims to locate the most relevant time segments from videos based on a given query, enabling users to retrieve content segments of interest more conveniently and quickly.

[0003] Existing video localization tasks can be divided into two types: (1) text-video localization and (2) speech-video localization. Most existing methods mainly address text-based tasks (1), while recent work has begun to study speech-based tasks (2). However, research on methods (2) still relies heavily on speech annotation, requiring annotators to accurately annotate each video segment with appropriate speech. Speech annotation requires significantly more time and manpower than text annotation, resulting in an expensive and tedious annotation burden. Therefore, to reduce the annotation burden and data costs, researching speech-video localization under weak supervision has become a solution. Under weak supervision, precise time annotation is lacking, and only coarse-grained speech-video correspondences exist. Therefore, how to efficiently model speech-video interaction content becomes the key to solving the problem. Summary of the Invention

[0004] To effectively perform voice-video interaction under weak supervision, this invention proposes a weakly supervised voice-video localization method and system based on semantic interaction.

[0005] The specific technical solution adopted in this invention is as follows:

[0006] In a first aspect, this invention proposes a weakly supervised speech-video localization method based on semantic interaction, comprising the following steps:

[0007] Step 1: Obtain a pre-trained dataset consisting of speech and corresponding text; extract the time-domain and frequency-domain features of the speech; then use a speech encoder to encode the time-domain and frequency-domain features to obtain speech embedding feature vectors; and extract text embedding feature vectors; the speech embedding feature vectors include acoustic feature vectors and semantic feature vectors.

[0008] Step 2: Perform acoustic-semantic pre-training based on speech embedding feature vectors and text embedding feature vectors. Based on the one-to-one semantic relationship between speech and text, obtain loss function terms for neural network-based temporal classification, information-noise contrast estimation, negative log-likelihood, and similarity constraint, and pre-train the speech encoder together.

[0009] Step 3: Obtain the localization task dataset consisting of speech and corresponding video, forming positive and negative sample pairs; for each sample pair, extract the semantic feature vectors corresponding to the time-domain and frequency-domain features of the speech, concatenate them to obtain the final speech semantic features; and extract the video embedding feature vectors.

[0010] Step 4: Using the final speech semantic features and video embedding feature vectors of the sample pairs as input, the localization network is used to predict the candidate segment with the highest score in each sample pair. The highest score is used as the alignment score, and the binary cross-entropy loss function term is calculated based on the alignment score.

[0011] Step 5: Using the candidate segments with the highest scores of the positive sample pairs predicted by the localization network, construct frame-level positive and negative samples, and obtain a loss function term based on maximizing mutual information through acoustic-visual contrastive learning.

[0012] Step 6: Combine the loss function terms obtained in Step 4 and Step 5 to obtain the final complete loss function, and perform weakly supervised localization training on the localization network.

[0013] Step 7: Obtain the speech and video to be localized, and use the speech encoder and video encoder to obtain the final speech semantic features and video embedding feature vectors respectively; interact the two, fuse each candidate segment with the speech, calculate the score of each candidate segment through the localization network, and output the candidate segment with the highest score to complete the localization.

[0014] Furthermore, the speech encoder includes several acoustic coding layers and several semantic coding layers. Both the acoustic coding layers and the semantic coding layers are composed of linear layers, batch normalization, ReLU activation functions, and residual connections. Taking the time-domain features or frequency-domain features of speech as input, the time-domain features or frequency-domain features are first encoded by several acoustic coding layers to generate acoustic feature vectors. Then, the acoustic feature vectors are encoded by several semantic coding layers to generate semantic feature vectors corresponding to the time-domain features or frequency-domain features.

[0015] Furthermore, in step 4, a top-down approach is adopted. First, candidate segments are generated based on the video, and then the similarity score between each candidate segment and the speech is calculated to predict the candidate segment with the highest score in each sample pair.

[0016] Furthermore, in step 4, step 5) includes:

[0017] 5.1) Using the candidate segments with the highest scores of positive sample pairs predicted by the localization network, frame-level positive and negative samples are constructed using location-based and score-based methods;

[0018] 5.2) Using the mutual information estimation method based on Jensen-Shannon divergence, calculate the loss function term based on maximizing mutual information:

[0019]

[0020] in, Let sp(·) denote the loss function term based on maximizing mutual information, C(·) denote the softplus function, E[·] denote the mutual information calculation, and E[·] denote the expectation. Represents the final speech semantic features, v + v represents a positive sample at the frame level. - This represents a negative sample at the frame level.

[0021] Furthermore, the location-based method is as follows: based on the boundary of the candidate segment with the highest score, the frame features outside the boundary are used as negative samples, and the frame features inside the boundary are used as positive samples; the score-based method is as follows: using a localization network to obtain the scores of each frame within the boundary of the candidate segment with the highest score, and the frame features with the highest and lowest scores are selected as positive samples and negative samples, respectively.

[0022] Secondly, this invention proposes a weakly supervised speech-video localization system based on semantic interaction, comprising:

[0023] The pre-trained data module is used to acquire a pre-trained dataset consisting of speech and corresponding text.

[0024] The location task data module is used to acquire a location task dataset consisting of voice and corresponding video.

[0025] The speech encoder module is used to encode the time-domain and frequency-domain features of speech to obtain speech embedding feature vectors. The speech embedding feature vectors include acoustic feature vectors and semantic feature vectors. The result of concatenating the semantic feature vectors of the time-domain and frequency-domain features is used as the final speech semantic features.

[0026] The video encoder module is used to extract video embedding feature vectors.

[0027] The text encoder module is used to extract text embedding feature vectors;

[0028] The localization network module is used to interact with the final speech semantic features and video embedding feature vectors of the sample pairs, fuse each candidate segment with speech, and predict the candidate segment with the highest score in each sample pair.

[0029] The acoustic-semantic pre-training module is used to perform acoustic-semantic pre-training based on speech embedding feature vectors and text embedding feature vectors. Based on the one-to-one semantic relationship between speech and text, it obtains loss function terms based on neural network-based temporal classification, loss function terms based on information-noise contrast estimation, loss function terms based on negative log-likelihood, and loss function terms based on similarity constraints, which are used to pre-train the speech encoder.

[0030] The acoustic-visual contrastive learning module is used to construct positive and negative sample pairs based on the localization task dataset. The highest score output by the localization network module is used as the alignment score, and a binary cross-entropy loss function is calculated based on the alignment score. Furthermore, using candidate segments of the highest scores of the positive sample pairs predicted by the localization network, frame-level positive and negative samples are constructed. Through acoustic-visual contrastive learning, a loss function based on maximizing mutual information is obtained. Combining the binary cross-entropy loss function and the loss function based on maximizing mutual information, the localization network is trained under weak supervision.

[0031] Compared with the prior art, the beneficial effects of this invention are:

[0032] (1) By using speech-text pre-training, this invention enables the speech encoder to learn semantic information in speech information in the absence of fine-grained time annotation. Multiple pre-training tasks work together from acoustic and speech perspectives to achieve accurate and effective speech coding, thereby avoiding expensive annotation in speech-video localization and achieving efficient speech-video localization under weak supervision.

[0033] (2) Through speech-video contrast learning during weak supervision, this invention optimizes the correlation between speech and similar video frames directly from the representation space by acoustic-visual contrast learning in the absence of fine-grained time annotation. This allows the model to learn cross-modal semantic space similarity at the encoding level, which helps the model to better locate speech-related frames globally in the video and achieve efficient speech-video localization under weak supervision.

[0034] In summary, by combining speech-text pre-training and speech-video contrastive learning, this invention can fully extract useful semantic information from speech and achieve efficient cross-modal interaction, thereby replacing expensive fine-grained annotation and achieving efficient speech-video localization under weak supervision. Attached Figure Description

[0035] Figure 1This is a schematic diagram of a weakly supervised speech-video localization backbone network based on semantic interaction, as shown in an embodiment of the present invention.

[0036] Figure 2 This is a voice encoder shown in an embodiment of the present invention.

[0037] Figure 3 This is a schematic diagram of acoustic-semantic pre-training shown in an embodiment of the present invention.

[0038] Figure 4 This is a schematic diagram of acoustic-visual contrastive learning as shown in an embodiment of the present invention. Detailed Implementation

[0039] The present invention will be further described and illustrated below with reference to the accompanying drawings and specific embodiments.

[0040] This invention proposes a weakly supervised speech-video localization method based on semantic interaction, comprising the following steps:

[0041] Step 1: Obtain a pre-trained dataset consisting of speech and corresponding text; extract the temporal and frequency domain features of the speech; then encode the temporal and frequency domain features using a speech encoder to obtain the speech embedding feature vector; and extract the text embedding feature vector. In this embodiment, the speech embedding feature vector includes acoustic feature vectors and semantic feature vectors.

[0042] Step 2: Perform acoustic-semantic pre-training based on speech embedding feature vectors and text embedding feature vectors. Based on the one-to-one semantic relationship between speech and text, obtain loss function terms for neural network-based temporal classification, information-noise contrast estimation, negative log-likelihood, and similarity constraint, and pre-train the speech encoder together.

[0043] Step 3: Obtain the localization task dataset consisting of speech and corresponding video, forming positive and negative sample pairs; for each sample pair, extract the semantic feature vectors corresponding to the time-domain and frequency-domain features of the speech, and concatenate them to obtain the final speech semantic features. And extract video embedding feature vectors.

[0044] Step 4: Using the final speech semantic features and video embedding feature vectors of the sample pair as input, the localization network is used to predict the candidate segment with the highest score in each sample pair. The highest score is used as the alignment score, and the binary cross-entropy loss function term is calculated based on the alignment score.

[0045] Step 5: Using the candidate segments with the highest scores of the positive sample pairs predicted by the localization network, construct frame-level positive and negative samples, and obtain a loss function term based on maximizing mutual information through acoustic-visual contrastive learning.

[0046] Step 6: Combine the loss function terms obtained in Step 4 and Step 5 to obtain the final complete loss function, and perform weakly supervised localization training on the localization network.

[0047] Step 7: Obtain the speech to be localized and the video containing candidate segments. Use the speech encoder and video encoder to obtain the final speech semantic features and video embedding feature vectors, respectively. Interact the two and fuse each candidate segment with the speech. Calculate the score of each candidate segment through the localization network and output the candidate segment with the highest score to complete the localization.

[0048] Step 1 above, which uses a speech encoder to obtain speech embedding feature vectors, can be implemented in the following way:

[0049] For each given speech input S, dual encoding is performed on the speech features, while also considering the temporal features S. t and frequency domain features S f ;like Figure 2 As shown, for S t / S f , and pass it through N a An acoustic coding layer consisting of a linear layer, batch normalization, ReLU activation function, and residual connections is used to extract the acoustic feature vector S. a , will S a As the input to the acoustic encoder, after passing through N s A semantic coding layer identical to the acoustic coding layer is used to extract the semantic feature vector S. s For clarity, the time-domain feature S t and frequency domain features S f The corresponding semantic feature vectors are respectively represented as and The final speech semantic features are obtained after concatenation.

[0050] In step 1 above, the sub-word embedding vectors in the pre-trained word embedding model are used as text embedding feature vectors.

[0051] like Figure 3 As shown, the acoustic-semantic pre-training in step 2 above includes four collaborative tasks, which are implemented as follows:

[0052] Collaborative Task 1:

[0053] 2.1) Based on the one-to-one relationship between speech and text in the pre-training dataset, the acoustic feature vector S generated by the acoustic encoder... a Above, obtain the loss function term for time-series class classification based on neural networks:

[0054] 2.1.1) Using acoustic feature vector Sa As input, a decoder consisting of two layers of feedforward neural networks is used to generate prediction sequences, each of which represents a set of paths consisting of target text characters and "whitespace" symbols.

[0055] 2.1.2) Using the forward-backward dynamic programming algorithm, calculate the sum of probabilities of all possible paths in the predicted sequence. Taking the original text sequence as the target, optimize the sum of probabilities of all paths based on the forward and backward probabilities at any time, and obtain the loss function term for the time-series classification based on the neural network:

[0056]

[0057] in, This represents the loss function term for temporal class classification based on neural networks. C represents a set of paths corresponding to the sequence generated by the decoder. One of the paths, S a represents the acoustic feature vector, and P represents the probability that the two paths are the same.

[0058] Time-domain features S t and frequency domain features S f The corresponding loss function terms for neural network-based temporal class classification are denoted as follows: and

[0059] Collaborative Task 2:

[0060] 2.2) The semantic feature vector S generated by the semantic encoder s Above, the corresponding text embedding feature vector is used as the positive sample T. + And randomly sample several other text embedding feature vectors as negative samples T - Obtain the loss function term based on information noise contrast estimation.

[0061] 2.2.1) Using semantic feature vector S s As input, a bidirectional gated recurrent neural network and sequence-dimensional average pooling are used to obtain the global semantic feature vector S. g .

[0062] 2.2.2) Based on the one-to-one relationship between speech and text in the pre-training dataset, apply sequence-dimensional average pooling to the corresponding text embedding feature vectors to obtain global text feature vectors as positive samples T. + Similarly, several texts that do not correspond to the speech are randomly sampled from the pre-training dataset to obtain the global text feature vector as negative samples T. - .

[0063] 2.2.3) Calculate the loss function term based on information-noise contrast estimation:

[0064]

[0065] in, This represents the loss function term based on information noise contrast estimation, where B represents the number of negative samples sampled.

[0066] Time-domain features S t and frequency domain features S f The corresponding loss function terms based on information noise contrast estimation are denoted as follows: and

[0067] Collaborative Task 3:

[0068] 2.3) Transfer the semantic feature vector S s and the text embedding feature vector Q after randomly masking several characters m The input to the decoder is the predicted text character generated by the decoder; the target is the original text character, and a loss function term based on negative log-likelihood is obtained.

[0069] 2.3.1) Randomly mask the text embedding feature vector Q obtained in step 1 to obtain the text embedding feature vector Q after randomly masking several characters. m ; the semantic feature vector S s and the text embedding feature vector Q after randomly masking several characters m As input to the decoder, the masked text characters are predicted using a multi-head attention-based decoder.

[0070] 2.3.2) Calculate the loss function term based on negative log-likelihood based on the predicted text characters and the original text characters:

[0071]

[0072] in, Let n represent the loss function term based on negative log-likelihood. w w represents the number of text characters i+1 Represents the (i+1)th original text character, e i This represents the predicted i-th text character.

[0073] Time-domain features S t and frequency domain features S f The corresponding loss function terms based on negative log-likelihood are denoted as follows: and

[0074] Collaborative Task 4:

[0075] 2.4) Based on the time-domain characteristics S t and frequency domain features S f Corresponding semantic feature vector and Calculate the loss term based on similarity constraints:

[0076]

[0077] in, This represents the loss term based on similarity constraints.

[0078] In step 3, a localization task dataset consisting of speech and corresponding videos is obtained. Given an undressed video V and a speech query S, where a video can correspond to multiple speech queries but lacks temporal annotation, i.e., it is unknown which segment the speech describes. Following a multi-instance learning approach, unrelated videos V′ and speech queries S′ are sampled from the training set to form two unrelated speech-video pairs (V′, S) and (V, S′). For each sample pair, semantic feature vectors corresponding to the temporal and frequency domain features of the speech are extracted, and concatenated to obtain the final speech semantic features. (exist Figure 1 (referred to as speech features in Chinese); and, using a pre-trained video encoder to extract video embedding feature vectors (in... Figure 1 (hereinafter referred to as video features), and the video encoder here is a technology known in the field.

[0079] In step 4, the localization network is used to calculate the scores of each candidate segment in the video of each sample pair, and the highest candidate segment score is used as the alignment score. The localization network here is a well-known technology in the field. It calculates the scores of each candidate segment in the video by interacting with the final speech semantic features and video embedding feature vectors in the input speech-video pair, and the highest candidate segment score is used as the alignment score.

[0080] In this embodiment, a top-down approach is adopted, that is, firstly, candidate segments are generated based on the video, and then the similarity score between each candidate segment and the speech is calculated to select the most relevant candidate segment.

[0081] Based on the binary cross-entropy loss function, relevant speech-video pairs (V,S) are separated from unrelated speech-video pairs (V...S). ′ (,S) and (V,S) ′ The data is separated from the original data and subjected to weakly supervised speech-video localization training, using the binary cross-entropy loss function. as follows:

[0082]

[0083] Where f(k) represents the alignment score of the speech-video pair (V,S), f(k) v′ ) and f(k s′ ) represent voice-video pairs (V ′ (S) and voice-video pairs (V,S) ′ Alignment score. Employing multi-instance learning, the model distinguishes between related and unrelated speech-video pairs by randomly sampling unrelated video and speech pairs.

[0084] Step 5, using Figure 4 Acoustic-visual contrastive learning is used to obtain a loss function based on maximizing mutual information.

[0085] Reliable frame-level positive samples v are obtained by using the candidate segments with the highest scores of positive sample pairs predicted by the localization network and employing two screening methods. + and negative sample v - Used for characterization optimization.

[0086] The first location-based method directly selects frame features outside the predicted candidate segment boundaries as negative samples.

[0087] The second score-based method calculates the score of each frame within the candidate segment boundary, and selects the highest and lowest score frames as positive and negative samples, respectively. In this embodiment, the video frames within the candidate segment boundary and the final speech semantic features are used as input to the localization network to directly obtain the score of each frame.

[0088] Using a mutual information estimation method based on Jensen-Shannon divergence, the mutual information between the final speech semantic features and the frame-level positive / negative sample features is calculated. Comparatively, the mutual information between speech and positive sample frames is made greater than that between speech and negative sample frames. A loss function based on maximizing mutual information is then calculated.

[0089]

[0090] Where sp(·) represents the softplus function, C(·) represents mutual information calculation, and E[·] represents expectation. Represents the final speech semantic features, v + v represents a positive sample at the frame level. - This represents a negative sample at the frame level, where positive / negative samples represent frame-level features.

[0091] In step 6, the joint binary cross-entropy loss function is used. and loss function based on maximizing mutual information Weakly supervised localization training is performed on the localization network.

[0092] In summary, this invention first utilizes the weighted loss function results of the four collaborative tasks in step 2, and employs the gradient descent learning method to pre-train the parameters of the speech encoder. After pre-training, the speech encoder parameters are frozen. Then, using the weighted loss function results in steps 4 and 5, the gradient descent learning method is employed to train the parameters of the localization network, thereby completing the weakly supervised speech-video localization task.

[0093] In the actual localization process, speech and video containing candidate segments are acquired. The speech encoder and video encoder are used to obtain the final speech semantic features and video embedding feature vectors, respectively. The two interact to fuse each candidate segment with the speech. The localization network calculates the score of each candidate segment and outputs the candidate segment with the highest score to achieve localization.

[0094] The above method will be applied to the following embodiments to demonstrate the technical effects of the present invention. The specific steps in the embodiments will not be repeated.

[0095] This invention conducts experiments on weakly supervised speech-video localization on the ActivityNet Speech dataset and pre-trains on the Libri Speech dataset. To objectively evaluate the performance of this invention, three evaluation metrics—R@1,IoU=m, R@5,IoU=m, and mIoU—are used to evaluate its effectiveness on the selected test set (m=0.1 / 0.3 / 0.5 / 0.7). Here, R@n,IoU=m represents the value where the IoU between the n highest-scoring candidate segments and the ground truth labeled segments is greater than m, and mIoU represents the average IoU between the highest-scoring candidate segment and the ground truth labeled segments.

[0096] The method of this invention is compared with the following prior art models, specifically divided into two frameworks:

[0097] 1) Cascaded Framework: This framework first converts speech queries into recognized text queries using an automatic speech recognition model in the first stage. In the second stage, the text query is used as input to apply traditional text-video localization methods for training. In the first stage, an automatic speech recognition model based on wav2vec 2.0 is selected. In the second stage, the following text-video localization method is selected:

[0098] 1. The WSLLN model directly fuses candidate fragment features with text features and develops alignment and probe branches. Finally, it combines the output scores of the two branches to generate the final fragment alignment score.

[0099] 2. The RTBPN model generates enhanced and suppressed video features through a text-based filter, is trained using intra-sample and inter-sample loss functions, and employs regularization techniques to improve stability.

[0100] 3. The SCN model masks keywords in the text and uses a decoder to reconstruct the masked words based on visual fragment features. Finally, the candidate fragments are ranked according to the feedback scores.

[0101] 2) End-to-end framework: This framework directly uses voice queries as input for voice-video localization training. On one hand, it uses the backbone network Base as the basic localization network and combines it with different pre-training methods, including:

[0102] 1. The Wav2vec method uses contrastive predictive coding for pre-training, and it does not rely on the text information corresponding to the speech;

[0103] 2. The MMP method utilizes a unified multimodal encoder for pre-training on two sub-tasks: "masking-reconstruction" and "prediction consistency".

[0104] 3. The LUT method utilizes knowledge distillation to perform pre-training at both the overall sentence and local word levels.

[0105] On the other hand, the cooperative task 2 of the present invention is combined with different localization networks. Since there is currently no weakly supervised speech-video localization model, this embodiment uses speech input instead of text input in the previous weakly supervised text-video localization network as the localization network, including the WSLLN model and RTBPN model mentioned above.

[0106] The experimental results obtained by following the steps described in the specific implementation method are shown in Tables 1 to 3. The model of the present invention is represented as SIN.

[0107] Table 1: Results of this invention on the weakly supervised speech-video localization task on the ActivityNet Speech dataset (R@1 and mIoU)

[0108]

[0109] Table 2: Results of this invention on the weakly supervised speech-video localization task on the ActivityNet Speech dataset (R@5)

[0110]

[0111] As can be seen from Tables 1 and 2, the localization performance of the SIN method proposed in this invention is significantly better than other methods, such as the cascaded frame (ASR) RTBPN and the end-to-end frame RTBPN+CT. Compared with the baseline model Base, the SIN method of this invention improves R@1, IoU = 0.7 from 11.02 to 15.02, R@5, IoU = 0.7 from 28.08 to 35.48, and mIoU from 28.44 to 34.02.

[0112] Benefiting from the acoustic-semantic pre-training and acoustic-visual contrastive learning proposed in this invention, the SIN method can achieve efficient speech coding and efficient cross-modal interaction, ultimately achieving effective weakly supervised speech-video localization. The SIN method outperforms other methods, demonstrating that effectively extracting speech information and achieving cross-modal interaction are crucial for unannotated sign language translation tasks.

[0113] This embodiment also provides a weakly supervised voice-video localization system based on semantic interaction, which is used to implement the above embodiments. The terms "module," "unit," etc., used below refer to combinations of software and / or hardware that implement a predetermined function. Although the system described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible.

[0114] A weakly supervised speech-video localization system based on semantic interaction, comprising:

[0115] The pre-trained data module is used to acquire a pre-trained dataset consisting of speech and corresponding text.

[0116] The location task data module is used to acquire a location task dataset consisting of voice and corresponding video.

[0117] The speech encoder module is used to encode the time-domain and frequency-domain features of speech to obtain speech embedding feature vectors. The speech embedding feature vectors include acoustic feature vectors and semantic feature vectors. The result of concatenating the semantic feature vectors of the time-domain and frequency-domain features is used as the final speech semantic features.

[0118] The video encoder module is used to extract video embedding feature vectors.

[0119] The text encoder module is used to extract text embedding feature vectors;

[0120] The localization network module is used to interact with the final speech semantic features and video embedding feature vectors of the sample pairs, fuse each candidate segment with speech, and predict the candidate segment with the highest score in each sample pair.

[0121] The acoustic-semantic pre-training module is used to perform acoustic-semantic pre-training based on speech embedding feature vectors and text embedding feature vectors. Based on the one-to-one semantic relationship between speech and text, it obtains loss function terms based on neural network-based temporal classification, loss function terms based on information-noise contrast estimation, loss function terms based on negative log-likelihood, and loss function terms based on similarity constraints, which are used to pre-train the speech encoder.

[0122] The acoustic-visual contrastive learning module is used to construct positive and negative sample pairs based on the localization task dataset. The highest score output by the localization network module is used as the alignment score, and a binary cross-entropy loss function is calculated based on the alignment score. Furthermore, using candidate segments of the highest scores of the positive sample pairs predicted by the localization network, frame-level positive and negative samples are constructed. Through acoustic-visual contrastive learning, a loss function based on maximizing mutual information is obtained. Combining the binary cross-entropy loss function and the loss function based on maximizing mutual information, the localization network is trained under weak supervision.

[0123] For the system embodiments, since they basically correspond to the method embodiments, relevant details can be found in the descriptions of the method embodiments; the implementation methods of the remaining modules will not be repeated here. The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0124] The system embodiments of the present invention can be applied to any device with data processing capabilities, such as a computer or other similar device. The system embodiments can be implemented in software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor of any data processing device loading the corresponding computer program instructions from non-volatile memory into memory for execution.

[0125] The above examples are merely specific embodiments of the present invention. Obviously, the present invention is not limited to the above embodiments and many variations are possible. All variations that can be directly derived or conceived by those skilled in the art from the disclosure of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A weakly supervised speech-video localization method based on semantic interaction, characterized in that, Includes the following steps: Step 1: Obtain a pre-trained dataset consisting of speech and corresponding text; extract the time-domain and frequency-domain features of the speech; then use a speech encoder to encode the time-domain and frequency-domain features to obtain speech embedding feature vectors; and extract text embedding feature vectors; the speech embedding feature vectors include acoustic feature vectors and semantic feature vectors. Step 2: Perform acoustic-semantic pre-training based on speech embedding feature vectors and text embedding feature vectors. Based on the one-to-one semantic relationship between speech and text, obtain loss function terms for neural network-based temporal classification, information-noise contrast estimation, negative log-likelihood, and similarity constraint, and pre-train the speech encoder together. Step 3: Obtain the localization task dataset consisting of speech and corresponding video, forming positive and negative sample pairs; for each sample pair, extract the semantic feature vectors corresponding to the time domain features and frequency domain features of the speech, and concatenate them to obtain the final speech semantic features. And extracting video embedding feature vectors; Step 4: Using the final speech semantic features and video embedding feature vectors of the sample pairs as input, the localization network is used to predict the candidate segment with the highest score in each sample pair. The highest score is used as the alignment score, and the binary cross-entropy loss function term is calculated based on the alignment score. Step 5: Using the candidate segments with the highest scores of the positive sample pairs predicted by the localization network, construct frame-level positive and negative samples, and obtain a loss function term based on maximizing mutual information through acoustic-visual contrastive learning. Step 6: Combine the loss function terms obtained in Step 4 and Step 5 to obtain the final complete loss function, and perform weakly supervised localization training on the localization network. Step 7: Obtain the speech and video to be located, and use the speech encoder and video encoder to obtain the final speech semantic features and video embedding feature vectors respectively; The two interact, fusing each candidate segment with the speech, and calculates the score of each candidate segment through a localization network. The candidate segment with the highest score is then output to complete the localization.

2. The weakly supervised speech-video localization method based on semantic interaction according to claim 1, characterized in that, The speech encoder includes several acoustic coding layers and several semantic coding layers. Both the acoustic coding layers and the semantic coding layers are composed of linear layers, batch normalization, ReLU activation functions, and residual connections. Taking the time-domain features or frequency-domain features of speech as input, the time-domain features or frequency-domain features are first encoded by several acoustic coding layers to generate acoustic feature vectors. Then, the acoustic feature vectors are encoded by several semantic coding layers to generate semantic feature vectors corresponding to the time-domain features or frequency-domain features.

3. The weakly supervised speech-video localization method based on semantic interaction according to claim 1, characterized in that, The loss function term for the neural network-based temporal class classification is specifically as follows: 2.1.1) Using the acoustic feature vectors corresponding to the time-domain or frequency-domain features as input, a decoder composed of a two-layer feedforward neural network is used to generate a prediction sequence, where each sequence represents a set of paths consisting of target text characters and "whitespace" symbols; 2.1.2) Calculate the sum of probabilities of all possible paths in the predicted sequence. Using the original text sequence as the target, calculate the loss function term for time-series classification based on neural networks: in, This represents the loss function term for temporal class classification based on neural networks. C represents a set of paths corresponding to the sequence generated by the decoder. One of the paths, S a represents the acoustic feature vector, and P represents the probability that the two paths are the same.

4. The weakly supervised speech-video localization method based on semantic interaction according to claim 1, characterized in that, The loss function term based on information noise contrast estimation is specifically as follows: 2.2.1) Using the semantic feature vector corresponding to the time-domain or frequency-domain features as input, a bidirectional gated recurrent neural network and average pooling along the sequence dimension are used to obtain the global semantic feature vector S. g ; 2.2.2) Based on the one-to-one relationship between speech and text in the pre-training dataset, apply sequence-dimensional average pooling to the corresponding text embedding feature vectors to obtain global text feature vectors as positive samples T. + Randomly sample several texts that do not correspond to the speech from the pre-training dataset to obtain the global text feature vector as negative samples T. - ; 2.2.3) Calculate the loss function term based on information-noise contrast estimation: in, This represents the loss function term based on information noise contrast estimation, where B represents the number of negative samples sampled.

5. The weakly supervised speech-video localization method based on semantic interaction according to claim 1, characterized in that, The loss function term based on negative log-likelihood is specifically as follows: 2.3.1) Randomly mask the text embedding feature vector obtained in step 1 to obtain the text embedding feature vector Q after randomly masking several characters. m The semantic feature vector corresponding to the time-domain or frequency-domain features and the text embedding feature vector Q after randomly masking several characters are combined. m As input to the decoder, the masked text characters are predicted using a multi-head attention-based decoder. 2.3.2) Calculate the loss function term based on negative log-likelihood based on the predicted text characters and the original text characters: in, Let n represent the loss function term based on negative log-likelihood. w w represents the number of text characters i+1 Represents the (i+1)th original text character, e i This represents the predicted i-th text character.

6. The weakly supervised speech-video localization method based on semantic interaction according to claim 1, characterized in that, The loss function term based on similarity constraints is specifically as follows: Based on the semantic feature vectors corresponding to time-domain features and frequency-domain features and Calculate the loss term based on similarity constraints: in, This represents the loss term based on similarity constraints.

7. The weakly supervised speech-video localization method based on semantic interaction according to claim 1, characterized in that, In step 4, a top-down approach is adopted. First, candidate segments are generated based on the video. Then, the similarity score between each candidate segment and the speech is calculated to predict the candidate segment with the highest score in each sample pair.

8. The weakly supervised speech-video localization method based on semantic interaction according to claim 1, characterized in that, Step 5) in step 4 includes: 5.1) Using the candidate segments with the highest scores of positive sample pairs predicted by the localization network, frame-level positive and negative samples are constructed using location-based and score-based methods; 5.2) Using the mutual information estimation method based on Jensen-Shannon divergence, calculate the loss function term based on maximizing mutual information: in, Let sp(·) denote the loss function term based on maximizing mutual information, C(·) denote the softplus function, E[·] denote the mutual information calculation, and E[·] denote the expectation. Represents the final speech semantic features, v + v represents a positive sample at the frame level. - This represents a negative sample at the frame level.

9. A weakly supervised speech-video localization method based on semantic interaction according to claim 8, characterized in that, The location-based method is as follows: based on the boundary of the candidate segment with the highest score, the frame features outside the boundary are used as negative samples, and the frame features inside the boundary are used as positive samples; the score-based method is as follows: using a localization network to obtain the scores of each frame within the boundary of the candidate segment with the highest score, and the frame features with the highest and lowest scores are selected as positive samples and negative samples, respectively.

10. A weakly supervised speech-video localization system based on semantic interaction, characterized in that, include: The pre-trained data module is used to acquire a pre-trained dataset consisting of speech and corresponding text. The location task data module is used to acquire a location task dataset consisting of voice and corresponding video. The speech encoder module is used to encode the time-domain and frequency-domain features of speech to obtain speech embedding feature vectors. The speech embedding feature vectors include acoustic feature vectors and semantic feature vectors. The result of concatenating the semantic feature vectors of the time-domain and frequency-domain features is used as the final speech semantic features. The video encoder module is used to extract video embedding feature vectors. The text encoder module is used to extract text embedding feature vectors; The localization network module is used to interact with the final speech semantic features and video embedding feature vectors of the sample pairs, fuse each candidate segment with speech, and predict the candidate segment with the highest score in each sample pair. The acoustic-semantic pre-training module is used to perform acoustic-semantic pre-training based on speech embedding feature vectors and text embedding feature vectors. Based on the one-to-one semantic relationship between speech and text, it obtains loss function terms based on neural network-based temporal classification, loss function terms based on information-noise contrast estimation, loss function terms based on negative log-likelihood, and loss function terms based on similarity constraints, which are used to pre-train the speech encoder. The acoustic-visual contrastive learning module is used to construct positive and negative sample pairs based on the localization task dataset. The highest score output by the localization network module is used as the alignment score, and a binary cross-entropy loss function is calculated based on the alignment score. Furthermore, using candidate segments of the highest scores of the positive sample pairs predicted by the localization network, frame-level positive and negative samples are constructed. Through acoustic-visual contrastive learning, a loss function based on maximizing mutual information is obtained. Combining the binary cross-entropy loss function and the loss function based on maximizing mutual information, the localization network is trained under weak supervision.

Images