Video keyword retrieval method and device based on multiple encoders and electronic equipment

Abstract

The invention provides a video keyword retrieval method and device based on multiple encoders and electronic equipment. The method comprises the following steps: acquiring a lip region image sequence of a to-be-detected video; the lip region image sequence is input into a visual feature extraction network, the visual feature extraction network comprises multiple visual encoders arranged in parallel, different types of visual features are extracted and fused, and multi-encoder visual features are obtained; inputting the to-be-detected keyword into a text encoder, and extracting to obtain a text feature; and performing alignment processing on the text feature and the multi-encoder visual feature to generate a joint feature, inputting the joint feature into a classifier, and outputting the probability of occurrence of the to-be-detected keyword in the video. A multi-encoder structure is introduced to carry out modeling on video lip features, cross-modal alignment is realized in combination with text features, and the problem that keyword detection cannot be carried out in a noisy environment or a silent scene in the prior art is effectively solved.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence and multimodal information processing technology, and more specifically, to a video keyword retrieval method, apparatus, and electronic device based on a multi-encoder. Background Technology

[0002] With the development of deep learning, speech recognition and keyword retrieval technologies have been widely applied in scenarios such as intelligent interaction, audio and video retrieval, and content moderation. However, in noisy environments, long-distance acquisition, or silent video, audio signals are often difficult to obtain or of poor quality, making speech-based keyword retrieval methods ineffective. In contrast, lip-reading technology, by utilizing lip movements, mouth shape changes, and related facial muscle features in video, can still extract certain semantic information in silent or low signal-to-noise ratio environments, thus possessing significant application value in silent video surveillance, security review, and privacy scenarios.

[0003] Existing speech keyword retrieval methods mainly fall into three categories: sequence labeling methods based on fill-in-the-blank models, dynamic time warping methods based on example matching, and indexing methods that rely on large-vocabulary continuous speech recognition systems. However, these methods either have limited modeling capabilities, high computational complexity, or heavily depend on speech recognition performance, making them unsuitable for direct transfer to lip-reading keyword retrieval tasks. In the field of lip-reading recognition, research has largely focused on complete sentence-level or word-level recognition, with relatively insufficient research on keyword-level retrieval. This results in a lack of effective solutions in practical applications where only keyword extraction is required. Therefore, how to efficiently and accurately complete keyword retrieval using video information in the absence of audio has become a pressing technical problem that needs to be solved. Summary of the Invention

[0004] This disclosure provides at least one video keyword retrieval method, apparatus, and electronic device based on a multi-encoder architecture. It introduces a multi-encoder structure to model video lip features and combines text features to achieve cross-modal alignment, effectively solving the problem in the prior art that keyword detection is impossible in noisy or silent environments.

[0005] This disclosure provides a video keyword retrieval method based on multiple encoders, including:

[0006] Obtain the image sequence of the lip region from the video to be detected;

[0007] The image sequence of the lip region is input into a visual feature extraction network, which includes multiple visual encoders set in parallel to extract different types of visual features and fuse them to obtain multi-encoder visual features.

[0008] The keywords to be detected are input into the text encoder to extract text features;

[0009] The text features are aligned with the visual features of the multi-encoder to generate joint features, and the joint features are input into the classifier to output the probability of the keyword to be detected appearing in the video.

[0010] In one optional implementation, the method further includes:

[0011] Collect sample keywords, sample video data, and corresponding sample audio data;

[0012] Extract the lip region sequence from the sample video data and extract Mel frequency cepstral coefficient features from the sample audio data;

[0013] The sample lip region sequence is input into each of the visual encoders to determine the multi-encoder visual features, and the sample keywords are input into the text encoder to determine the text features;

[0014] The Mel frequency cepstral coefficient features are aligned with the text features and then input into the posterior encoder to determine the intermediate hidden layer features.

[0015] The posterior encoder is trained by aligning the visual features of the multi-encoder to the features of the intermediate hidden layer based on the KL divergence loss constraint.

[0016] In one optional implementation, the joint features are input into the classifier, specifically including:

[0017] The joint features are input into the trained posterior encoder, and the joint features are divided into a first joint feature part and a second joint feature part along the channel;

[0018] The first joint feature part is subjected to convolution and nonlinear processing, and then subtracted from the second joint feature part to determine the transformed second joint feature part;

[0019] The first joint feature part and the transformed second joint feature part are concatenated along the channel to obtain the posterior encoder output feature, and the posterior encoder output feature is input into the classifier.

[0020] In one optional implementation, the Mel frequency cepstral coefficient features are aligned with the text features and then input into the posterior encoder to determine the intermediate hidden layer features, specifically including:

[0021] The text-to-speech fusion feature, which is aligned with the Mel frequency cepstral coefficient feature and the text feature, is input into the a posteriori encoder, and the text-to-speech fusion feature is divided into a first text-to-speech feature part and a second text-to-speech feature part along the channel;

[0022] After performing convolution and nonlinear processing on the first textual feature part, it is added to the second textual feature part to determine the transformed second textual feature part;

[0023] The intermediate hidden layer features are output by concatenating the first textual feature portion with the transformed second textual feature portion along the channel.

[0024] In one optional implementation, the text features are aligned with the multi-encoder visual features based on the following formula:

[0025]

[0026] Where F represents the joint feature; cat represents concatenation along the sequence length; repeat represents repetition N times along the channel number, N = Cv / Ct; Ft represents the text feature, Ft ∈ [Tt, Ct]; Fv represents the video feature, Fv ∈ [Tv, Cv]; Tt represents the length of the text feature sequence, Ct represents the dimension of the text feature; Tv represents the length of the video feature sequence, Cv represents the dimension of the video feature.

[0027] In one optional implementation, the visual encoder includes at least Bi-LSTM, Transformer, and Conformer;

[0028] The classifier consists of a Bi-LSTM and a fully connected layer.

[0029] This disclosure also provides a video keyword retrieval device based on multiple encoders, the device comprising:

[0030] The video acquisition module is used to acquire the image sequence of the lip region of the video to be detected;

[0031] The visual feature extraction module is used to input the image sequence of the lip region into the visual feature extraction network. The visual feature extraction network includes multiple visual encoders set in parallel, which extract different types of visual features and fuse them to obtain multi-encoder visual features.

[0032] The text feature extraction module is used to input the keywords to be detected into the text encoder and extract the text features;

[0033] The retrieval module is used to align the text features with the visual features of the multi-encoder to generate joint features, and input the joint features into the classifier to output the probability of the keyword to be detected appearing in the video.

[0034] In one optional embodiment, the apparatus further includes a training module, the training module being used for:

[0035] Collect sample keywords, sample video data, and corresponding sample audio data;

[0036] Extract the lip region sequence from the sample video data and extract Mel frequency cepstral coefficient features from the sample audio data;

[0037] The sample lip region sequence is input into each of the visual encoders to determine the multi-encoder visual features, and the sample keywords are input into the text encoder to determine the text features;

[0038] The Mel frequency cepstral coefficient features are aligned with the text features and then input into the posterior encoder to determine the intermediate hidden layer features.

[0039] The posterior encoder is trained by aligning the visual features of the multi-encoder to the features of the intermediate hidden layer based on the KL divergence loss constraint.

[0040] This disclosure also provides an electronic device, including: a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the above-described video keyword retrieval method based on multiple encoders, or any possible implementation of the above-described video keyword retrieval method based on multiple encoders.

[0041] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of the above-described video keyword retrieval method based on multiple encoders, or any possible implementation thereof.

[0042] This disclosure also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the above-described video keyword retrieval method based on multiple encoders, or the steps in any possible implementation of the above-described video keyword retrieval method based on multiple encoders.

[0043] This disclosure provides a video keyword retrieval method, apparatus, and electronic device based on a multi-encoder architecture. The method involves acquiring a sequence of lip region images from a video to be detected; inputting the lip region image sequence into a visual feature extraction network, which includes multiple visual encoders configured in parallel to extract different types of visual features and fuse them to obtain multi-encoder visual features; inputting the keyword to be detected into a text encoder to extract text features; aligning the text features with the multi-encoder visual features to generate joint features; and inputting the joint features into a classifier to output the probability of the keyword appearing in the video. By introducing a multi-encoder structure to model the lip features of the video and combining them with text features to achieve cross-modal alignment, this effectively solves the problem in existing technologies where keyword detection is impossible in noisy or silent environments.

[0044] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the embodiments will be briefly described below. These drawings are incorporated in and constitute a part of this specification. They illustrate embodiments conforming to this disclosure and, together with the specification, serve to explain the technical solutions of this disclosure. It should be understood that the following drawings only show some embodiments of this disclosure and should not be considered as limiting the scope. Those skilled in the art can obtain other related drawings based on these drawings without creative effort.

[0046] Figure 1 A flowchart of a video keyword retrieval method based on multiple encoders provided in this disclosure is shown.

[0047] Figure 2 A flowchart illustrating a training method for a video keyword retrieval model provided in an embodiment of this disclosure is shown;

[0048] Figure 3 A schematic diagram of a video keyword retrieval device based on a multi-encoder provided in an embodiment of this disclosure is shown;

[0049] Figure 4 A schematic diagram of an electronic device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0051] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0052] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0053] Research has revealed that existing speech keyword retrieval methods mainly fall into three categories: sequence labeling methods based on fill-in-the-blank models, dynamic time warping methods based on example matching, and indexing methods that rely on large-vocabulary continuous speech recognition systems. However, these methods either have limited modeling capabilities, high computational complexity, or heavily depend on speech recognition performance, making them unsuitable for direct transfer to lip-reading keyword retrieval tasks. In the field of lip-reading recognition, research has largely focused on complete sentence-level or word-level recognition, with relatively insufficient research on keyword-level retrieval. This results in a lack of effective solutions for practical applications requiring only keyword extraction. Therefore, how to efficiently and accurately complete keyword retrieval using video information in the absence of audio has become a pressing technical problem that needs to be solved.

[0054] Based on the above research, this disclosure provides a video keyword retrieval method, apparatus, and electronic device based on multiple encoders. The method involves acquiring a sequence of lip region images from a video to be detected; inputting the lip region image sequence into a visual feature extraction network, which includes multiple visual encoders configured in parallel to extract different types of visual features and fuse them to obtain multi-encoder visual features; inputting the keyword to be detected into a text encoder to extract text features; aligning the text features with the multi-encoder visual features to generate joint features; and inputting the joint features into a classifier to output the probability of the keyword appearing in the video. By introducing a multi-encoder structure to model video lip features and combining them with text features to achieve cross-modal alignment, this effectively solves the problem of keyword detection in noisy or silent environments in existing technologies.

[0055] To facilitate understanding of this embodiment, a detailed description of the video keyword retrieval method based on multiple encoders disclosed in this disclosure is provided first. The execution entity of the video keyword retrieval method based on multiple encoders provided in this disclosure is generally a computer device with certain computing capabilities. This computer device may include, for example, a terminal device, a server, or other processing devices. The terminal device may be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, etc. In some possible implementations, this video keyword retrieval method based on multiple encoders can be implemented by a processor calling computer-readable instructions stored in memory.

[0056] See Figure 1 The diagram shows a flowchart of a video keyword retrieval method based on multiple encoders provided in this disclosure. The method includes steps S101 to S104, wherein:

[0057] S101. Obtain the image sequence of the lip region of the video to be detected.

[0058] In one embodiment of the present invention, for the video to be detected, it is first necessary to obtain the image sequence of the lip region in the video that is most directly related to the speaker's semantic expression.

[0059] Specifically, after inputting a video, the system first uses a face detection algorithm to locate the face region in the video frame. This detection algorithm can use a face detection model based on a convolutional neural network, such as MTCNN or RetinaFace, so as to accurately capture the face position in complex backgrounds and under different lighting conditions.

[0060] Here, after the face region is detected, a facial landmark detection algorithm is further used to determine the precise location of the lips. This algorithm is typically based on a deep regression model and can identify multiple key points, including the corners of the mouth and the lip line. Through the geometric relationships of these key points, this invention crops the region of interest (ROI) for the lips from the face region and performs size normalization and image enhancement to ensure the consistency and robustness of the input data in subsequent feature extraction processes.

[0061] In the process of processing video frame sequences, this invention also performs time series modeling on consecutive frames, that is, arranges the cropped lip ROIs in order according to the sampling rate of the video to form image sequence data of the lip region.

[0062] It should be noted that, in order to further improve the stability of the data, the system will perform face alignment and standardization on the cropped lip sequence to avoid interference caused by changes in head posture, camera angle or distance.

[0063] Meanwhile, in order to reduce redundant information and improve the efficiency of model training and inference, the number of sequence frames can be balanced during the preprocessing stage, for example, by using fixed frame rate sampling or time interpolation methods, to ensure that the length of the input sequence is adapted to the model requirements.

[0064] Thus, through the above steps, the present invention can reliably extract lip region image sequences reflecting semantic information from the original video and provide high-quality input data for subsequent multi-encoder visual feature extraction.

[0065] S102. The image sequence of the lip region is input into a visual feature extraction network, which includes multiple visual encoders set in parallel to extract different types of visual features and fuse them to obtain multi-encoder visual features.

[0066] In one embodiment of the present invention, the acquired image sequence of the lip region is input into a visual feature extraction network. This visual feature extraction network does not rely on a single path, but rather is a parallel structure composed of multiple visual encoders. Each encoder, upon receiving the same sequence, processes and represents the input data from different modeling perspectives. This avoids the limitation of a single network's ability to model complex data, and fully leverages the complementarity of different structures, improving the expressiveness and robustness of the final output features.

[0067] Here, this invention specifically selects three core encoders: Bidirectional Long Short-Term Memory (Bi-LSTM), Transformer based on self-attention mechanism, and Conformer combining convolution and attention mechanisms. Bi-LSTM can fully exploit the dynamic features of lip movements over time, effectively representing the temporal dependencies of continuous mouth shape changes; Transformer, through multi-head self-attention, can capture long-distance dependencies within a sequence globally, thus effectively modeling complex contextual information; and Conformer combines the local pattern extraction capability of convolution with the global modeling capability of attention, enabling it to simultaneously focus on detailed changes and overall trends in scenarios with long input sequences.

[0068] In this process, each lip region image sequence is fed into various visual encoders for feature extraction. Different encoders focus on different aspects of the input sequence in the feature space: Bi-LSTM prioritizes extracting continuously changing features over time, Transformer excels at global correlation modeling, and Conformer further enhances its ability to extract local detail features. Through a parallel structure, this invention can simultaneously retain feature representations from different encoders and fuse these multiple features at the feature output. The fusion method can be point-by-point addition, concatenation, or weighted averaging. This embodiment uses point-by-point addition, ensuring that the features at each time step simultaneously contain multiple representations from different encoders, thus forming multi-encoder visual features.

[0069] Specifically, the Bi-LSTM encoder employs a bidirectional long short-term memory (LSTM) structure, its core being the capture of temporal correlations in the input sequence through memory and forget gate mechanisms. Since lip movements often exhibit sequential dependencies—for example, the pronunciation of a word is composed of multiple consecutive mouth shapes—Bi-LSTM can simultaneously model from both forward and backward directions, extracting dynamic features across frames. These temporal features can comprehensively characterize the continuity of the speaker's mouth shape changes, helping to distinguish lip movement patterns corresponding to different keywords.

[0070] Here, the Transformer encoder relies on a self-attention mechanism to establish the association between any two positions in the input sequence. By calculating the similarity between the query vector and the key vector, the Transformer can globally determine the relationship between the lip state of a frame and other frames in the sequence, thereby capturing long-distance dependency information. In keyword retrieval tasks, some lip features are not only reflected in a single frame but also need to be inferred from the context. The Transformer shows a significant advantage in this type of modeling. In addition, the parallel computing capability of the Transformer also greatly improves the efficiency of feature extraction, enabling the system to adapt to the real-time requirements of real-world scenarios while maintaining accuracy.

[0071] Here, the Conformer encoder combines the advantages of convolutional neural networks and attention mechanisms. Convolutional modules excel at extracting fine-grained edge, contour, and motion variation features within a local scope, while multi-head self-attention mechanisms are adept at capturing long-range dependencies. Conformer organically combines the two, first aggregating local spatiotemporal features through convolutional modules, then introducing global information through the attention mechanism, and finally introducing feedforward modules and normalization layers on this basis. This allows it to preserve both the fine local structure and global semantic consistency when modeling long sequences. This structure is particularly effective for lip region image sequences because changes in lip shape include both subtle lip line movements and continuous overall deformation; the combination of both is necessary to comprehensively reflect the lip features of keywords.

[0072] In this way, by using this multi-encoder fusion approach, the visual feature extraction network can avoid the feature loss problem that is prone to occur in single models, thus improving the discriminative ability in video keyword retrieval tasks. Experiments show that this multi-encoder structure can significantly improve retrieval accuracy on the same dataset, especially maintaining high stability in noisy environments or low-resolution video conditions. This method not only ensures comprehensive modeling of lip movements but also provides high-quality input data for subsequent alignment and classification with text features.

[0073] S103. Input the keyword to be detected into the text encoder to extract the text features.

[0074] In one embodiment of the present invention, the keywords to be detected are input into a text encoder to extract the corresponding text features. Since text and video differ in modality, the text encoder transforms the keyword sequence in natural language form into a dense vector representation, enabling it to be aligned and fused with the visual features of multiple encoders in the video within a unified feature space.

[0075] Specifically, the input keywords to be detected are first processed through word segmentation and embedding, transforming the natural language text into a sequence of word vectors. Word vectors can be obtained based on pre-trained embedding models, such as word2vec, GloVe, or more advanced context-based dynamic embedding models, to ensure that words with similar semantics have close representations in the vector space.

[0076] In a preferred embodiment of the present invention, a 256-dimensional embedding vector is used to represent keywords, thereby reducing computational complexity while maintaining strong semantic distinctiveness.

[0077] Here, after obtaining the word vector sequence, it is input into the text encoder for deep modeling. This invention uses a gated recurrent unit (GRU) as the basic structure of the text encoder. In sequence modeling, the GRU effectively solves the gradient vanishing and gradient exploding problems that occur in traditional recurrent neural networks (RNNs) during long sequence training by alternating control of update and reset gates. For keyword text, although the sequence length is usually not long, the GRU can capture the dependencies between words while maintaining computational efficiency, thus obtaining text feature representations with temporal information and semantic context.

[0078] During the encoding process, GRU progressively updates the state of the word vector sequence. The final output vector not only contains the overall semantics of the keywords but also retains their contextual position information within the sequence. In this way, the present invention can obtain text features with uniform dimensions and complete semantics, and further normalize and align them for concatenation or fusion with visual features from multiple encoders in subsequent stages.

[0079] Thus, through the above steps, the text encoder of the present invention can efficiently convert natural language keywords into a representation that matches visual features, thereby achieving effective connection of cross-modal features.

[0080] S104. Align the text features with the multi-encoder visual features to generate joint features, and input the joint features into the classifier to output the probability of the keyword to be detected appearing in the video.

[0081] In one embodiment of the present invention, to achieve cross-modal information fusion, it is necessary to align the text features obtained by the text encoder with the multi-encoder visual features generated by the visual feature extraction network to generate a unified joint feature representation. Since text and video have inherent differences in sequence length and feature dimensions, direct splicing or fusion can easily lead to information loss or semantic misalignment. Therefore, the present invention designs a specific mapping and repetition mechanism in the feature alignment stage.

[0082] Specifically, let the text features be Ft∈[Tt,Ct], where Tt represents the length of the text sequence and Ct represents the dimension of the text features; and let the visual features be Fv∈[Tv,Cv], where Tv represents the length of the video sequence and Cv represents the dimension of the visual features. This invention repeatedly expands the text features along the channel number direction, ensuring that the text features maintain the same dimension as the visual features. Then, through a concatenation operation along the sequence length direction, the two are combined into a new joint feature F, expressed by the formula:

[0083]

[0084] Where F represents joint features; cat represents concatenation along the sequence length; repeat represents repetition N times along the channel number, N = Cv / Ct; Ft represents text features, Ft ∈ [Tt, Ct]; Fv represents video features, Fv ∈ [Tv, Cv]; Tt represents the length of the text feature sequence, Ct represents the dimension of the text features; Tv represents the length of the video feature sequence, Cv represents the dimension of the video features. Through the above processing, text features and visual features are aligned and jointly represented in dimensions, enabling the model to simultaneously capture semantic information and lip-sync dynamics features.

[0085] After obtaining the joint features, this invention inputs them into a classifier for discrimination. The classifier consists of a Bidirectional Long Short-Term Memory (Bi-LSTM) network and fully connected layers. The Bi-LSTM can further perform temporal modeling on the joint feature sequence, extracting the interaction relationship between text and video in the temporal dimension, thereby improving the accuracy of keyword detection. Subsequently, the fully connected layers map the high-dimensional features output by the Bi-LSTM to a probability space, outputting the probability of the keyword appearing in the video sequence. This probability not only reflects the presence or absence of the keyword but also, through its numerical value, reflects the model's confidence in the discrimination result.

[0086] Thus, through the above processing flow, this invention achieves effective alignment and fusion of cross-modal features, enabling textual semantic information and visual dynamic features to be collaboratively modeled in the same representation space. Ultimately, the probability results output by the classifier accurately reflect the occurrence of keywords in the video, maintaining high robustness and retrieval accuracy even in noisy or audio-free scenarios, thereby meeting the practical application requirements for rapid keyword detection and localization.

[0087] Further, see Figure 2 The diagram shows a flowchart of a training method for a video keyword retrieval model provided in this embodiment of the present disclosure. The method includes steps S201 to S205, wherein:

[0088] S201. Collect sample keywords, sample video data, and corresponding sample audio data.

[0089] S202. Extract the sample lip region sequence from the sample video data and extract the Mel frequency cepstral coefficient features from the sample audio data.

[0090] S203. Input the sample lip region sequence into each of the visual encoders to determine the multi-encoder visual features, and input the sample keywords into the text encoder to determine the text features.

[0091] S204. Align the Mel frequency cepstral coefficient features with the text features and input them into the posterior encoder to determine the intermediate hidden layer features.

[0092] S205. Based on the KL divergence loss constraint, align the visual features of the multi-encoder to the features of the intermediate hidden layer, and train the posterior encoder.

[0093] In one embodiment of the present invention, in order to improve the stability and accuracy of the model when relying solely on video features during the inference phase, audio features are introduced as auxiliary supervision signals in addition to video and text data during the training process, thereby achieving alignment and constraint of visual features and audio features through a posterior encoder.

[0094] Specifically, sample keywords, sample video data, and corresponding sample audio data are first collected in advance. The sample keywords can be predefined common words or phrases, such as "technological innovation" or "library." The video data includes diverse video clips, both containing and not containing these keywords, and the audio data consists of speech signals acquired synchronously during video capture. In the preprocessing stage, for the sample video data, face detection and keypoint localization algorithms are used to extract the corresponding lip region sequences, forming a time-series input of lip movements. Simultaneously, feature engineering is performed on the sample audio data to extract Mel-frequency cepstral coefficients (MFCC) features, converting the audio signal into a dense feature representation suitable for deep learning model processing.

[0095] Here, in the feature extraction stage, the lip region sequence is input into multiple parallel visual encoders, including Bi-LSTM, Transformer, and Conformer. Each encoder independently extracts visual features, which are then fused to form multi-encoder visual features. Simultaneously, sample keywords are input into a text encoder to obtain text features that match the dimensions of the visual features. Subsequently, the MFCC audio features and text features are aligned and fused dimensionally to form text-speech fusion features, which are then input into a posterior encoder for processing. The posterior encoder employs a normalized flow-based structure, consisting of multiple stacked coupling layers, which progressively maps the text-speech fusion features to a complex latent distribution, outputting intermediate hidden layer features. These intermediate features act as a bridge between audio and vision, guiding the learning direction of visual features.

[0096] In the training and optimization process, this invention not only utilizes the cross-entropy loss function to supervise the classification results but also introduces Kullback-Leibler (KL) divergence loss to constrain the alignment of visual features from the multi-encoder to the intermediate hidden layer features output by the posterior encoder. Through this constraint, the model can continuously adjust the parameters of the visual encoder during training, gradually approximating the distribution provided by audio supervision, thus effectively compensating for the instability of modeling solely based on video features. Ultimately, driven by the joint loss function, the posterior encoder is fully trained, enabling the entire system to achieve high-precision keyword retrieval even in the absence of audio information during the inference phase, relying solely on video features.

[0097] As one possible implementation, during inference, the joint features are input into the trained posterior encoder, and the joint features are divided into a first joint feature part and a second joint feature part along the channel; the first joint feature part is subjected to convolution and nonlinear processing and then subtracted from the second joint feature part to determine the transformed second joint feature part; the first joint feature part and the transformed second joint feature part are concatenated along the channel to obtain the output features of the posterior encoder, and the output features of the posterior encoder are input into the classifier.

[0098] In one embodiment of the present invention, the aligned joint features are further input into a pre-trained posterior encoder to enhance feature modeling capabilities and improve classification performance. The joint features are first divided into a first joint feature part and a second joint feature part along the channel dimension. The first part and the second part are pre-set in terms of quantity and dimension so that they can be operated on in corresponding ways in subsequent calculations.

[0099] In the processing, the posterior encoder first applies a convolution operation to the first joint feature part to extract its local pattern information and spatial correlation. Then, it performs nonlinear activation processing on the convolution result to give the feature representation stronger nonlinear representation capabilities. The first joint feature part, after convolution and nonlinear processing, is then subtracted point-by-point from the original second joint feature part to obtain the transformed second joint feature part. In this way, the model can establish a difference mapping relationship between the first and second feature parts, thereby improving the overall feature discriminative ability.

[0100] After the above steps are completed, the original first joint feature part and the transformed second joint feature part are concatenated along the channel dimension to obtain a new composite feature representation, which is the output feature of the posterior encoder. This output feature not only integrates the difference information after convolution and nonlinear mapping, but also maintains the temporal and spatial continuity of the original features, and can more comprehensively characterize the potential distribution structure of the input joint features.

[0101] Finally, the features output by the posterior encoder are fed into the classifier for discrimination. The classifier consists of a bidirectional long short-term memory network (Bi-LSTM) and fully connected layers. The Bi-LSTM further performs temporal modeling on the feature sequence output by the posterior encoder to capture the contextual dependencies of the joint features in the temporal dimension. The fully connected layers are responsible for mapping the high-dimensional temporal features to the probability space, outputting the probability value of the target keyword appearing in the video sequence. This probability not only reflects whether the keyword appears in the video, but also reflects the model's confidence in the retrieval results through its numerical value, thus providing a reliable basis for video keyword detection in practical applications.

[0102] As another possible implementation, during training, the text-to-speech fusion features, which are aligned with the Mel frequency cepstral coefficient features and the text features, are input into the posterior encoder. The text-to-speech fusion features are divided into a first text-to-speech feature part and a second text-to-speech feature part along the channel. The first text-to-speech feature part is convolved and nonlinearly processed, and then added to the second text-to-speech feature part to determine the transformed second text-to-speech feature part. The first text-to-speech feature part and the transformed second text-to-speech feature part are concatenated along the channel to output the intermediate hidden layer features.

[0103] In one embodiment of the present invention, in order to effectively utilize audio features to assist in the supervision of visual features during the training phase, the Mel-frequency cepstral coefficients (MFCC) features extracted from the audio source are dimensionally aligned with the text features output by the text encoder to obtain text-speech fusion features. Text-speech fusion features can simultaneously contain the time-frequency distribution information carried by the audio signal and the semantic information of the text in the same representation space, thereby providing richer supervision signals for subsequent feature modeling.

[0104] In the posterior encoder process, the text-to-speech fusion features are first divided into a first text-to-speech feature part and a second text-to-speech feature part along the channel dimension. The first text-to-speech feature part mainly serves as the basic input for the difference transformation, while the second text-to-speech feature part is used as the component to be updated. Subsequently, a convolution operation is applied to the first text-to-speech feature part to capture its local feature patterns, and then a nonlinear activation function is used to process the convolution result, thereby enhancing the nonlinear expressive power of the features. The processed first text-to-speech feature part and the second text-to-speech feature part are added point by point to obtain the transformed second text-to-speech feature part. Through this "additive coupling" method, the model can introduce new difference information while maintaining the stability of the original features, avoiding excessive disruption of the input feature distribution.

[0105] After the above transformation, the first textual feature portion and the transformed second textual feature portion are concatenated along the channel dimension to obtain a new composite feature representation. This composite feature is the output of the posterior encoder and is provided as an intermediate hidden layer feature for subsequent training. During the training phase, this invention uses Kullback-Leibler divergence (KL divergence) loss to constrain the feature distribution of the visual multi-encoder output to gradually approximate the distribution of this intermediate hidden layer feature, thereby enabling the visual features to maintain discriminative ability consistent with audio even in the inference phase where audio input is lacking.

[0106] Through the above steps, this invention utilizes the processing of audio-text fusion features in the posterior encoder to achieve deep integration of audio and text features, and enhances feature modeling capabilities through a reversible mapping structure. The final output of intermediate hidden layer features effectively improves the system's convergence speed and retrieval accuracy during training.

[0107] This disclosure provides a video keyword retrieval method based on multiple encoders. The method involves acquiring a sequence of lip region images from a video to be detected; inputting the lip region image sequence into a visual feature extraction network, which includes multiple visual encoders configured in parallel to extract different types of visual features and fuse them to obtain multi-encoder visual features; inputting the keyword to be detected into a text encoder to extract text features; aligning the text features with the multi-encoder visual features to generate joint features; and inputting the joint features into a classifier to output the probability of the keyword appearing in the video. By introducing a multi-encoder structure to model video lip features and combining them with text features to achieve cross-modal alignment, this method effectively solves the problem of keyword detection in noisy or silent environments in existing technologies.

[0108] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0109] Based on the same inventive concept, this disclosure also provides a video keyword retrieval device based on multiple encoders, which corresponds to the video keyword retrieval method based on multiple encoders. Since the principle of the device in this disclosure is similar to the video keyword retrieval method based on multiple encoders described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0110] Please see Figure 3 , Figure 3 This is a schematic diagram of a video keyword retrieval device based on a multi-encoder, provided as an embodiment of this disclosure. Figure 3 As shown in the figure, the video keyword retrieval device 300 based on multiple encoders provided in this embodiment includes:

[0111] The video acquisition module 310 is used to acquire the image sequence of the lip region of the video to be detected.

[0112] The visual feature extraction module 320 is used to input the image sequence of the lip region into the visual feature extraction network. The visual feature extraction network includes multiple visual encoders set in parallel, which extract different types of visual features and fuse them to obtain multi-encoder visual features.

[0113] The text feature extraction module 330 is used to input the keywords to be detected into the text encoder and extract the text features.

[0114] The retrieval module 340 is used to align the text features with the visual features of the multi-encoder to generate joint features, and input the joint features into the classifier to output the probability of the keyword to be detected appearing in the video.

[0115] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0116] This disclosure provides a video keyword retrieval device based on a multi-encoder architecture. The device acquires a sequence of lip region images from a video to be detected. The lip region image sequence is input into a visual feature extraction network, which includes multiple visual encoders configured in parallel. These encoders extract different types of visual features and fuse them to obtain multi-encoder visual features. The keyword to be detected is input into a text encoder to extract text features. The text features are aligned with the multi-encoder visual features to generate joint features. These joint features are then input into a classifier to output the probability of the keyword appearing in the video. By introducing a multi-encoder structure to model the lip features of the video and combining them with text features to achieve cross-modal alignment, this effectively solves the problem in existing technologies where keyword detection is impossible in noisy or silent environments.

[0117] Corresponding to Figure 1 The present disclosure also provides an electronic device 400, such as a video keyword retrieval method based on multiple encoders. Figure 4 The diagram shown is a structural schematic of an electronic device 400 provided in an embodiment of this disclosure, including:

[0118] Processor 41, memory 42, and bus 43; memory 42 is used to store execution instructions, including main memory 421 and external memory 422; the main memory 421, also called internal memory, is used to temporarily store the computational data in processor 41, as well as the data exchanged with external memory 422 such as hard disk. Processor 41 exchanges data with external memory 422 through main memory 421. When the electronic device 400 is running, processor 41 and memory 42 communicate through bus 43, enabling processor 41 to execute... Figure 1 The steps of the video keyword retrieval method based on multiple encoders.

[0119] This disclosure also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program performs the steps of the video keyword retrieval method based on multiple encoders described in the above method embodiments. The storage medium can be a volatile or non-volatile computer-readable storage medium.

[0120] This disclosure also provides a computer program product, which includes computer instructions. When the computer instructions are executed by a processor, they can perform the steps of the video keyword retrieval method based on multiple encoders described in the above method embodiments. For details, please refer to the above method embodiments, which will not be repeated here.

[0121] The aforementioned computer program product can be implemented through hardware, software, or a combination thereof. In one optional embodiment, the computer program product is specifically embodied in a computer storage medium; in another optional embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0122] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the several embodiments provided in this disclosure, it should be understood that the disclosed device and method can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces; the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0123] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0124] In addition, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0125] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0126] Finally, it should be noted that the above-described embodiments are merely specific implementations of this disclosure, used to illustrate the technical solutions of this disclosure, and not to limit it. The protection scope of this disclosure is not limited thereto. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features, within the scope of the technology disclosed in this disclosure. Such modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be covered within the protection scope of this disclosure. Therefore, the protection scope of this disclosure should be determined by the protection scope of the claims.

Claims

1. A video keyword retrieval method based on multiple encoders, characterized in that, include: Obtain the image sequence of the lip region from the video to be detected; The image sequence of the lip region is input into a visual feature extraction network, which includes multiple visual encoders set in parallel to extract different types of visual features and fuse them to obtain multi-encoder visual features. The keywords to be detected are input into the text encoder to extract text features; The text features are aligned with the visual features of the multi-encoder to generate joint features, and the joint features are input into the classifier to output the probability of the keyword to be detected appearing in the video.

2. The method according to claim 1, characterized in that, The method further includes: Collect sample keywords, sample video data, and corresponding sample audio data; Extract the lip region sequence from the sample video data and extract Mel frequency cepstral coefficient features from the sample audio data; The sample lip region sequence is input into each of the visual encoders to determine the multi-encoder visual features, and the sample keywords are input into the text encoder to determine the text features; The Mel frequency cepstral coefficient features are aligned with the text features and then input into the posterior encoder to determine the intermediate hidden layer features. The posterior encoder is trained by aligning the visual features of the multi-encoder to the features of the intermediate hidden layer based on the KL divergence loss constraint.

3. The method according to claim 2, characterized in that, The joint features are input into the classifier, specifically including: The joint features are input into the trained posterior encoder, and the joint features are divided into a first joint feature part and a second joint feature part along the channel; The first joint feature part is subjected to convolution and nonlinear processing, and then subtracted from the second joint feature part to determine the transformed second joint feature part; The first joint feature part and the transformed second joint feature part are concatenated along the channel to obtain the posterior encoder output feature, and the posterior encoder output feature is input into the classifier.

4. The method according to claim 2, characterized in that, After aligning the Mel-frequency cepstral coefficient features with the text features, the data is input into the posterior encoder to determine the intermediate hidden layer features, specifically including: The text-to-speech fusion feature, which is aligned with the Mel frequency cepstral coefficient feature and the text feature, is input into the a posteriori encoder, and the text-to-speech fusion feature is divided into a first text-to-speech feature part and a second text-to-speech feature part along the channel; After performing convolution and nonlinear processing on the first textual feature part, it is added to the second textual feature part to determine the transformed second textual feature part; The intermediate hidden layer features are output by concatenating the first textual feature portion with the transformed second textual feature portion along the channel.

5. The method according to claim 1, characterized in that, The text features are aligned with the multi-encoder visual features based on the following formula: Where F represents the joint feature; cat represents concatenation along the sequence length; repeat represents repetition N times along the channel number, N = Cv / Ct; Ft represents the text feature, Ft ∈ [Tt, Ct]; Fv represents the video feature, Fv ∈ [Tv, Cv]; Tt represents the length of the text feature sequence, Ct represents the dimension of the text feature; Tv represents the length of the video feature sequence, Cv represents the dimension of the video feature.

6. The method according to claim 1, characterized in that: The visual encoder includes at least Bi-LSTM, Transformer, and Conformer; The classifier consists of a Bi-LSTM and a fully connected layer.

7. A video keyword retrieval device based on multiple encoders, characterized in that, include: The video acquisition module is used to acquire the lip region image sequence of the video to be detected; The visual feature extraction module is used to input the image sequence of the lip region into the visual feature extraction network. The visual feature extraction network includes multiple visual encoders set in parallel, which extract different types of visual features and fuse them to obtain multi-encoder visual features. The text feature extraction module is used to input the keywords to be detected into the text encoder and extract the text features; The retrieval module is used to align the text features with the visual features of the multi-encoder to generate joint features, and input the joint features into the classifier to output the probability of the keyword to be detected appearing in the video.

8. The apparatus according to claim 7, characterized in that, The device further includes a training module, the training module being used for: Collect sample keywords, sample video data, and corresponding sample audio data; Extract the lip region sequence from the sample video data and extract Mel frequency cepstral coefficient features from the sample audio data; The sample lip region sequence is input into each of the visual encoders to determine the multi-encoder visual features, and the sample keywords are input into the text encoder to determine the text features; The Mel frequency cepstral coefficient features are aligned with the text features and then input into the posterior encoder to determine the intermediate hidden layer features. The posterior encoder is trained by aligning the visual features of the multi-encoder to the features of the intermediate hidden layer based on the KL divergence loss constraint.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a bus. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory via the bus. When the machine-readable instructions are executed by the processor, they perform the steps of the video keyword retrieval method based on multiple encoders as described in any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the video keyword retrieval method based on any one of claims 1 to 6.

Images