A cross-language audio-video keyword matching method and system

Abstract

The application discloses a cross-language audio-video keyword matching method and system, and belongs to the technical field of computer language recognition. The method comprises the following steps: preprocessing original audio-video data to obtain standardized video frames, pure human audio and text metadata with timestamps; using a transformer model trained by contrast learning to map user keywords to a cross-language shared semantic space to obtain text features; extracting video spatiotemporal composite features and audio acoustic fingerprint features; taking the text features as a query, splicing audio-video features as keys and values, generating fusion features through cross attention, and compensating fusion weights according to signal-to-noise ratio changes and motion saliency nonlinearity; constructing a semantic association matrix with time sequence mask and sparsification constraints based on the fusion features, positioning candidate segments through a sliding window, and then performing bidirectional verification containing cross-language knowledge graph common sense verification to output a matching result. The application realizes millisecond-level accurate positioning of cross-language audio-video content.

Description

Technical Field

[0001] This application belongs to the field of computer language recognition technology, specifically relating to a cross-language audio and video keyword matching method and system. Background Technology

[0002] With the explosive growth of global multimedia data, intelligent retrieval and management of audio and video content has become a core research direction in the field of information technology. In the era of big data, multimodal information processing technologies covering voice, images, and text play an irreplaceable role in improving information retrieval efficiency and ensuring smooth cross-border communication. These technologies typically involve the storage, parsing, and feature mapping of large-scale unstructured data, aiming to achieve accurate classification and rapid location of massive multimedia resources, thereby building a bridge connecting digital content and users' semantic needs.

[0003] Among them, cross-language audio and video keyword matching technology integrates natural language processing and multimodal perception capabilities to achieve accurate content retrieval in different language environments. This technology mainly focuses on semantically aligning text query requests in a specific language with audio and video resources containing multilingual speech streams or subtitle streams. Its core objective is to eliminate language barriers and improve the searchability of audio and video content, providing key support for smart education, cross-border media monitoring, and global content distribution.

[0004] However, existing technologies mostly employ a cascaded approach of translation followed by matching. Due to limitations in translation accuracy, this approach is prone to semantic loss and error accumulation, and struggles to capture the deeper meaning of non-standardized language in audio and video. Furthermore, traditional matching methods often lack the ability to deeply integrate the spatiotemporal features of audio and video with the semantic features of the text, resulting in poor robustness of the retrieval system in complex acoustic environments or low-resolution visual scenes. In addition, existing cross-modal alignment algorithms are computationally expensive on large-scale datasets and have coarse-grained matching, failing to achieve millisecond-level localization and high-precision verification of key audio and video segments. Therefore, a cross-language audio and video keyword matching scheme is desired. Summary of the Invention

[0005] The purpose of this invention is to provide a cross-language audio and video keyword matching method and system, which can effectively solve the problems in the background art mentioned above.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A cross-language audio and video keyword matching method includes the following specific steps: The input cross-language audio and video raw data is preprocessed to obtain standardized video frame sequences, clean human voice audio streams, and time-stamped text metadata; Using a pre-trained model based on a transformer architecture, the keywords input by the user are mapped to a cross-language shared semantic representation space to obtain text semantic feature vectors. The model uses semantically identical sentence pairs and unrelated sentence pairs from a multilingual parallel corpus as samples, and is trained by a contrastive learning objective function to make the vector distance between semantically identical cross-language sentence pairs in the space less than a first preset threshold. Video spatiotemporal composite features are extracted based on the standardized video frame sequence, and audio acoustic fingerprint features are extracted based on the pure human voice audio stream. Using the text semantic feature vector as the query vector, and the concatenation of the video spatiotemporal composite feature and the audio acoustic fingerprint feature as the key vector and value vector, cross-attention calculation is performed to generate fusion features; wherein, based on the real-time signal-to-noise ratio change slope and the motion saliency of video keyframes, the fusion weights are adjusted through nonlinear compensation. A cross-modal semantic association matrix is ​​constructed based on the fusion features. During construction, a temporal causal mask is introduced, and sparsity constraints are applied by combining the topological features of cross-language grammatical structures. Based on the constrained matrix, candidate segments are located through a sliding window, and bidirectional verification is performed to generate matching results. The bidirectional verification includes the introduction of cross-language knowledge graphs for common sense logic verification in the backward consistency test.

[0007] Furthermore, the training of the pre-trained model includes: Positive sample sentence pairs are constructed by extracting semantically identical source language sentences and target language sentences from parallel corpora of multiple languages, and semantically unrelated sentence pairs are used as negative samples. The contrastive learning objective function aims to minimize the contrastive loss value, thereby maximizing the cosine similarity of positive sample sentence pairs and minimizing the cosine similarity of negative sample sentence pairs. During training, a Euclidean distance constraint is also introduced to ensure that the Euclidean distance between semantically similar cross-linguistic words in the vector space is less than the first preset threshold.

[0008] Furthermore, mapping user-input keywords to a cross-linguistic shared semantic representation space includes: A context-aware operator is introduced. This operator uses positional encoding information and the context vector output by the masked language model to perform weighted fusion with the static word vector of the keyword based on the semantic environment of the keyword in the query statement, so as to dynamically adjust the vector position of the keyword in the cross-language shared semantic representation space.

[0009] Furthermore, extracting video spatiotemporal composite features based on the standardized video frame sequence includes: The standardized video frame sequence is processed using a three-dimensional convolutional neural network, wherein the convolution kernel of the three-dimensional convolutional neural network performs convolution operations synchronously in the temporal and spatial dimensions. The output of the three-dimensional convolutional neural network is fed into a long short-term memory network or a gated recurrent unit to capture scene context, object category and action information along the time axis; A fixed-dimensional spatiotemporal composite feature vector for video is obtained by compression through a global average pooling layer.

[0010] Furthermore, extracting audio acoustic fingerprint features based on the pure human voice audio stream includes: The pure human voice audio stream is subjected to frame segmentation and windowing processing; The power spectrum is obtained by performing a fast Fourier transform on the framed audio signal, and the power spectrum is then passed through a set of Mel filter banks to obtain the logarithmic energy distribution of each frequency band. The discrete cosine transform is applied to extract the Mel frequency cepstral coefficients, which are used as the audio acoustic fingerprint features.

[0011] Furthermore, performing cross-attention calculations includes: The text semantic feature vector is passed through a fully connected linear transformation layer to obtain the query vector; The spliced ​​video spatiotemporal composite features and the audio acoustic fingerprint features are respectively passed through two independent linear transformation layers to obtain key vectors and value vectors; Calculate the scaled dot product of the query vector and the key vector, and obtain the attention weight map by normalization using a flexible maximum function; The attention weight map is applied to the value vector to generate the fused feature.

[0012] Furthermore, based on the slope of the real-time signal-to-noise ratio change and the motion saliency of video keyframes, the fusion weights are adjusted through nonlinear compensation, including: When the slope of the signal-to-noise ratio indicates that the noise intensity continues to increase within a preset time window, and the motion significance of the current video keyframe exceeds a preset motion threshold, the weight of the video spatiotemporal composite features during fusion is further increased through a nonlinear compensation function.

[0013] Furthermore, performing two-way verification on the candidate fragments includes: Forward matching verification: Extract the original speech data corresponding to the candidate segment, transcribe it into text through an automatic speech recognition model, and perform object detection on key video frames to identify visual tags, thereby constructing a semantic content description of the segment; Backward consistency test: Using a pre-trained cross-language similarity model, the cosine similarity between the semantic content description and the user's original query request is calculated to obtain the backward score; and a cross-language knowledge graph is introduced to perform common sense logic verification on the visual tags and the keywords of the user query, and the backward score is corrected based on the verification results. The cumulative matching score of the candidate fragment is calculated as the forward score. The forward score and the corrected backward score are weighted and averaged to obtain the final confidence score.

[0014] Furthermore, locating candidate segments using a sliding window includes: An adaptive sliding window is used to search the cross-modal semantic association matrix after the sparsification constraint. The length of the adaptive sliding window is dynamically adjusted according to the number of characters in the query text, and adjacent windows have a preset overlap rate. Calculate the cumulative matching score within the coverage area of ​​each window, and select the window with the highest score as the candidate segment.

[0015] A cross-language audio and video keyword matching system includes: The preprocessing module is used to preprocess the input cross-language audio and video raw data to obtain standardized video frame sequences, clean human voice audio streams, and timestamped text metadata. The semantic representation module is used to map user-input keywords to a cross-language shared semantic representation space using a pre-trained model based on a transformer architecture, thereby obtaining text semantic feature vectors. The model uses semantically identical sentence pairs and unrelated sentence pairs from a multilingual parallel corpus as samples, and is trained by a contrastive learning objective function to make the vector distance between semantically identical cross-language sentence pairs in the space less than a first preset threshold. The feature extraction module is used to extract video spatiotemporal composite features based on the standardized video frame sequence and to extract audio acoustic fingerprint features based on the pure human voice audio stream. The cross-modal fusion and matching module is used to generate fusion features by performing cross-attention calculation using the text semantic feature vector as the query vector and the concatenation of the video spatiotemporal composite features and audio acoustic fingerprint features as the key vector and value vector; wherein, the fusion weights are adjusted by nonlinear compensation based on the slope of the real-time signal-to-noise ratio change and the motion saliency of the video keyframes. The localization and verification module is used to construct a cross-modal semantic association matrix based on the fusion features. During construction, a temporal causal mask is introduced, and sparsity constraints are applied by combining the topological features of cross-language grammatical structures. Based on the constrained matrix, candidate segments are located through a sliding window, and bidirectional verification is performed to generate matching results. The bidirectional verification includes the introduction of cross-language knowledge graphs for common sense logic verification in the backward consistency test.

[0016] In summary, this application includes at least one of the following beneficial technical effects: 1. By constructing a cross-linguistic shared semantic representation space with predetermined dimensions, this invention abandons the traditional cascaded model of translation followed by matching, and directly realizes the association between different languages ​​at the semantic vector level. This mechanism fundamentally avoids the inevitable semantic loss, grammatical deviation, and proper noun translation errors in machine translation, significantly improving the semantic accuracy of cross-linguistic retrieval, especially demonstrating strong robustness when dealing with non-standardized spoken language and slang.

[0017] 2. This invention achieves deep coupling of audio, video, and text features through a multimodal cross-attention mechanism. Compared to traditional methods that rely solely on speech or text recognition, this invention can simultaneously capture visual cues in video and acoustic fingerprints in audio, maintaining extremely high matching accuracy even in complex acoustic environments or scenarios with severe visual interference. Through a positioning algorithm with predetermined precision, it achieves accurate extraction of key segments, controlling the positioning error within a preset range, greatly improving the granularity of audio and video content retrieval.

[0018] 3. Utilizing a distributed index database and feature compression technology, this invention achieves extremely short initial screening time when processing large-scale, massive datasets, and the overall matching process meets real-time processing requirements. The system undergoes deep optimization of hardware resources, enabling multi-channel concurrent monitoring with the support of high-performance graphics processors, thus meeting the urgent needs of real-time processing of large-scale audio and video data in fields such as smart education and cross-border media supervision.

[0019] 4. This invention introduces human voice separation, image super-resolution enhancement, and user feedback self-learning modules, enabling the system to cope with disadvantageous input conditions such as low resolution and high noise. Through continuous online incremental training, the system can automatically optimize the matching strategy according to actual application scenarios. As usage time increases, both matching accuracy and coverage show a continuous upward trend, demonstrating extremely high industrial application value and technological leadership. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the overall technical solution for cross-language audio and video keyword matching methods; Figure 2 This is a schematic diagram illustrating the core principle of cross-modal feature deep fusion and alignment; Figure 3 This is a flowchart illustrating the logical process of extracting spatiotemporal features from audio and video content and semantic features from text. Figure 4 This is a flowchart of the logic for the pre-precision positioning and two-way verification mechanism for keywords. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the following description is provided in conjunction with the appendix. Figure 1 -Appendix Figure 4 The present invention will be further described in detail below with reference to specific embodiments.

[0022] Firstly, the cross-language audio-visual keyword matching method disclosed in this application is implemented according to the following steps: The first step, S1, involves acquiring and preprocessing the input cross-language audio and video raw data. The purpose of step S1 is to convert audio and video streams of different formats, encoding methods, and language tracks into a unified, standardized data format suitable for subsequent deep neural network processing. Preprocessing covers three modalities: video frames, audio streams, and subtitles or metadata text, and is implemented through the following sub-steps.

[0023] Step S101: Data Stream Acquisition and Access; The system reads locally stored raw audio and video data through standardized multimedia communication protocols, such as RTSP, RTMP, or HLS, or through a file system interface. This raw data typically contains multiple language audio tracks, such as parallel or independent audio tracks in Chinese, English, French, etc., and may also contain embedded or external subtitle streams and high-resolution visual images.

[0024] During the access phase, the system establishes a multi-threaded concurrent processing mechanism, allocating independent buffers and demultiplexing modules for each audio and video stream. The demultiplexing module decomposes the composite media stream into video bitstreams, audio sampling sequences, and text metadata carrying time information. In this way, the system can process multiple inputs simultaneously and prepare for subsequent standardization processing.

[0025] Step S102: Video frame sequence normalization processing; For the decomposed video bitstream, the system calls a hardware-accelerated decoder for decoding, such as an NVDEC component based on a high-performance graphics processor, to convert the compressed encoding format into an uncompressed raw image sequence.

[0026] The extracted video frames are resampled according to a pre-set frame rate, such as 25 frames per second or 30 frames per second, to ensure continuity in the time dimension.

[0027] In the spatial dimension, a center cropping operation is performed on each frame of video image to remove any black borders or redundant information that may exist at the image edges. Subsequently, a bilinear interpolation algorithm is used to uniformly scale the image to a preset pixel specification, such as 224 pixels × 224 pixels or 299 pixels × 299 pixels, to ensure that its spatial resolution is fully adapted to the input layer requirements of the subsequent deep convolutional neural network.

[0028] In addition, the system performs histogram equalization on each frame of video. This process enhances the visibility of details in dark areas of the scene by redistributing the image's brightness histogram, thereby improving the contrast of visual features.

[0029] Step S103: Audio stream preprocessing and enhancement; For the extracted original audio sample sequence, the system first performs channel merging and resampling to convert it into a mono pulse code modulation audio stream with a preset sampling frequency, for example, 16000 Hz. To preserve speech details and balance computational load, the quantization precision of the audio stream is set to a predetermined quantization bit width, for example, 16-bit depth.

[0030] Subsequently, an adaptive gain control algorithm is used to dynamically compress and adjust the gain of the audio stream. Specifically, the short-time energy distribution of the audio signal is monitored in real time, and the amplification factor is automatically adjusted so that speech segments of different volumes can be maintained within a preset amplitude range.

[0031] Next, a deep clustering algorithm is introduced to perform the voice separation logic. This algorithm utilizes a pre-trained masking network to separate environmental background noise, such as traffic noise, wind noise, and background music, from the target human voice in the time-frequency domain. The system retains only the extracted pure human voice audio stream and sends it to the subsequent feature extraction module, thereby achieving a high matching recall rate in noisy environments.

[0032] The training process for the masking network is as follows: Training data construction: Collect a large number of mixed audio samples, each sample consisting of a pure target human voice audio stream and different types of background noise mixed at different signal-to-noise ratios. At the same time, prepare corresponding ideal binary masks or ideal ratio masks as labels, with the labels representing the proportion of the target human voice in the time-frequency unit.

[0033] Loss function design: The mean squared error loss function or the binary cross-entropy loss function is used to calculate the difference between the mask predicted by the network and the ideal mask. In addition, signal distortion ratio loss can be introduced to directly measure the waveform difference between the separated human voice and the clean human voice audio stream.

[0034] Training objective: To minimize the error between the predicted mask and the ideal mask, so that the network can accurately estimate the probability of each time-frequency unit belonging to the target human voice from the mixed audio, thereby generating a masking matrix to separate the target human voice from the background.

[0035] Through the above training, the masking network has the ability to extract pure human voices in complex acoustic environments.

[0036] Step S104: Standardize timestamp text information; The system extracts the subtitle text or associated metadata files embedded in the video and converts them into Unicode format using character encoding conversion technology. For each piece of text information, the corresponding start and end timestamps are extracted and encapsulated into a structured metadata object, thereby establishing a precise mapping index between text content and audio / video timeline.

[0037] In summary, step S1, through multi-threaded access and demultiplexing, video frame resampling and standardization, audio gain and voice separation, and subtitle text timestamp alignment, transforms the original multi-format audio and video data into a unified image sequence, a clean audio stream, and timestamped text metadata suitable for deep neural network input. This preprocessing stage significantly reduces interference from subsequent environmental noise, resolution differences, and format inconsistencies, providing a reliable data foundation for constructing a cross-language shared semantic representation space and achieving cross-modal feature fusion.

[0038] Step S2 involves constructing a cross-language shared semantic representation space, mapping keywords from any language input by the user into this space. This ensures that texts with the same semantics but different languages ​​are close in distance within the space, thus providing a unified semantic benchmark for subsequent cross-modal matching with audio and video content. This is achieved through the following sub-steps.

[0039] Step S201: Construction of a large-scale pre-trained model; The system utilizes a multilingual parallel corpus containing hundreds of millions of words to pre-train a deep neural network model based on the Transformer architecture. This corpus covers aligned sentences in over 100 different languages, such as Chinese-English parallel sentence pairs, French-German parallel sentence pairs, etc. The model contains a predetermined number of encoder layers, such as 12 or 24 layers. Each encoder layer integrates a predetermined number of attention heads, such as 12 or 16, to capture long-distance dependencies across languages ​​through a multi-head self-attention mechanism. The vocabulary size is set to a predetermined size of commonly used cross-lingual lexical units, such as 250,000 lexical units, to ensure coverage of common expressions in most languages.

[0040] The training details for this pre-trained model are as follows: Training data construction: Sentence pairs are extracted from a multilingual parallel corpus. Each sentence pair consists of a source language sentence and a target language sentence, and both are semantically identical. Simultaneously, semantically unrelated sentence pairs are randomly selected from different corpora as negative samples. During training, each sentence pair is used as input and labeled as semantically aligned.

[0041] Loss function design: A contrastive learning objective function is adopted, specifically in the form of: ; Where: Lcontrast Indicates the comparative loss value; h i A vector representation of a sentence in the source language; A vector representation of a target language sentence that has the same semantics as a source language sentence; A vector representation of a negative sample sentence that is semantically unrelated to the source language sentence; τ represents the cosine similarity function, used to calculate the degree of similarity between two vectors; τ represents the temperature coefficient, used to control the smoothness of the similarity distribution, with a default value of 0.05 or 0.1.

[0042] Training objective: To minimize the contrastive loss function, maximizing the cosine similarity of semantically identical cross-linguistic sentence pairs in the vector space, while minimizing the cosine similarity of semantically unrelated sentence pairs. Through this training, the model learns to map identical semantics from different languages ​​to nearby vector positions.

[0043] Step S202: Unified Vector Space Mapping; Input text query requests from different languages ​​into the pre-trained model described above. The model converts the input text sequence into an initial vector representation through an embedding layer, and after multiple nonlinear transformations, finally maps the semantic information of different languages ​​into a unified high-dimensional continuous vector space. The dimension of this space is a preset value, such as 768 or 1024 dimensions.

[0044] During training, in addition to the contrastive learning loss mentioned above, a distance constraint is introduced: ensuring that the Euclidean distance between semantically similar cross-linguistic words in the vector space is less than a first preset threshold, such as 0.5. Specifically, an Euclidean distance penalty term is added to the contrastive loss, and a penalty is applied when the distance between positive sample pairs exceeds the threshold. This achieves semantic consistency across language barriers.

[0045] Step S203: Semantic Ambiguity Handling and Context Awareness; In the process of constructing a shared semantic representation space, a context-aware operator is introduced. This operator dynamically adjusts the position of keywords in the vector space based on the semantic environment before and after the keywords in the query statement.

[0046] For example, the word "apple" might refer to a fruit entity or a technology company entity in different contexts. Context-aware operators use positional encoding information and context vectors output by masked language models to generate dynamic vector representations of the word that include contextual information.

[0047] In practice, the entire query statement is input into the pre-trained model, the multi-layer feature vectors corresponding to the target words are extracted, and they are weighted and fused with the static word vectors of the words. The fusion weights are automatically calculated based on the semantic relevance of the context.

[0048] In this way, the same word is mapped to different regions in the vector space under different contexts, thereby eliminating ambiguity. The semantic matching accuracy after disambiguation can be improved by a predetermined percentage compared to not using context awareness, for example, by 10% to 20%.

[0049] In summary, step S2 constructs a high-dimensional cross-lingual shared semantic representation space by pre-training a multilingual transformer model and employing a contrastive learning strategy. This space can map texts in different languages ​​but with the same semantics to similar vector positions, while also handling polysemy through context-aware operators. After this step, the cross-lingual keywords input by the user have been transformed into semantic vectors that the system can process uniformly.

[0050] Regarding step S3, after constructing the cross-language shared semantic representation space in step S2, the system can already convert keywords in any language input by the user into a unified semantic vector. However, audio and video content itself contains information from multiple modalities, such as visual and acoustic features, requiring the extraction of spatiotemporal and acoustic features that can match the aforementioned semantic vectors. This step extracts the corresponding deep feature representations for the preprocessed video frame sequence, audio stream, and user-input text query, respectively. This is specifically achieved through the following sub-steps.

[0051] Step S301: Video spatiotemporal composite feature extraction; The system uses a three-dimensional convolutional neural network to process the preprocessed video frame sequence generated in step S102. This network can adopt a C3D or I3D architecture, and the convolution kernel size is preset to 3×3×3, that is, convolution operations are performed simultaneously in three dimensions: time axis and spatial plane, so as to capture the spatial structure and temporal dynamic changes in the video at the same time.

[0052] The network contains a predetermined number of residual connection blocks, which are specifically responsible for extracting spatial object features from video frames, such as faces, object categories, and scene layouts. To further capture motion trajectory features over longer periods, the output feature tensors of the 3D convolutional layers are fed into a Long Short-Term Memory (LSTM) network or a gated recurrent unit (GRU).

[0053] LSTM or GRU recursively processes the feature sequence along the time axis, outputting a composite feature tensor containing scene context, object category, and action information. Finally, a global average pooling layer compresses the spatial and temporal dimensions of this tensor to obtain a fixed-dimensional feature vector, such as 512-dimensional one.

[0054] The training details for 3D convolutional neural networks and LSTM networks are as follows: Training data construction: Large-scale video action recognition datasets are used, such as Kinetics-400 or Smething-Smething. Each sample is a video clip, approximately 10 seconds long, labeled with a corresponding action category, such as running or playing ball. During training, the video frame sequence is used as input, and the corresponding action category is used as the supervision signal.

[0055] Loss function design: The cross-entropy loss function is adopted, and its form is: ; Where: L video y represents the cross-entropy loss value of the video feature extraction network; C represents the total number of action categories; c This represents the value of the one-hot encoded vector of the real label in the c-th category, which can be 0 or 1. This represents the probability that the network predicts the input video belongs to the c-th category.

[0056] Training objective: To minimize the cross-entropy loss, enabling the network to extract spatiotemporal features from video frame sequences that effectively distinguish different actions and object scenes. After training, the final classification layer is removed, and the output of the intermediate layers is used as the spatiotemporal composite feature vector of the video.

[0057] Step S302: Audio acoustic fingerprint feature extraction; The system performs frame segmentation processing on the clean audio stream obtained after preprocessing in step S103. The length of each frame is preset to 25 milliseconds, and the frame shift step size is preset to 10 milliseconds, so that there is overlap between adjacent frames, ensuring continuity on the time axis. A Hamming window is applied to each frame of audio signal to reduce spectral leakage effects.

[0058] The time-domain signal is then converted into a power spectrum using a Fast Fourier Transform (FFT). The power spectrum is then passed through a predetermined number of Mel filter banks, such as 40 or 80 filters, to obtain the logarithmic energy distribution in each frequency band.

[0059] Finally, Discrete Cosine Transform (DCT) is applied to extract Mel-frequency cepstral coefficients (MFCCs), with the output dimension preset to 13 or 40 dimensions. These coefficients constitute the acoustic fingerprint of the audio stream, representing the underlying cepstral features of speech.

[0060] Step S303: Deep semantic feature extraction of text; The system receives cross-language text keyword query statements input by the user. First, word segmentation is performed to obtain a word sequence. Positional encoding technology is used to assign a unique position vector to each position in the sequence to preserve the word order information in the sentence.

[0061] Subsequently, the system reuses the pre-trained Transfrmer model corresponding to the cross-language shared semantic representation space constructed in step S2. The lexical sequence with positional encoding is input into the encoder part of the model, and the global dependencies within the sequence are calculated through a multi-head self-attention mechanism.

[0062] The encoder outputs a deep semantic feature vector for each word in the current query. This vector not only contains the meaning of the word itself but also incorporates the logical structure information of the entire sentence. The training method, loss function, and training data of this pre-trained Transfrmer model have been described in detail in step S201 and will not be repeated here.

[0063] In summary, step S3 extracts video spatiotemporal composite features and audio acoustic fingerprint features from the preprocessed audio and video data, and simultaneously extracts deep semantic features from the user-input text query. These features reside in different modal spaces and have different dimensions and physical meanings.

[0064] To enable them to perform similarity comparisons and matching, the next step is to map these heterogeneous features into the same semantic space and perform deep fusion, which is to enter the cross-modal feature deep fusion and alignment process described in step S4.

[0065] Regarding step S4, in step S3, the system extracted video spatiotemporal composite features, audio acoustic fingerprint features, and text deep semantic features. These features reside in different modal spaces, with significant differences in dimensionality and numerical distribution, making direct similarity comparison impossible. The core task of this step is to map these heterogeneous features into the same semantic space and achieve deep fusion through an attention mechanism, thereby obtaining the matching probability between the audio / video content and the query text at each time point. This is specifically achieved through the following sub-steps.

[0066] Step S401: Multimodal cross-attention calculation; The system adopts a multimodal cross-attention mechanism, using text semantic feature vectors as guidance, to perform weighted fusion of audio and video composite features. The specific calculation process is as follows.

[0067] First, the deep semantic feature vector of the text output in step S303 is passed through a fully connected linear transformation layer to obtain the query vector, denoted as Q.

[0068] The video spatiotemporal composite feature tensor output in step S301 and the audio acoustic fingerprint feature tensor output in step S302 are concatenated on the time axis and then passed through two independent linear transformation layers to obtain the key vector K and value vector V, respectively. The weight matrix of the above linear transformation layer is obtained through training.

[0069] Then, the dot product of the query vector and the key vector is calculated and divided by the square root of the key vector's dimension. Scaling is applied to measure the correlation between text semantics and audio / video content at various time steps. The attention weight map is obtained by normalization using the Softmax function.

[0070] Finally, this weight map is applied to the value vector to generate enhanced cross-modal fusion features. The mathematical expression of this process is as follows: ; Where: Attention(Q,K,V) represents the output cross-modal fusion feature vector; Q represents the query vector obtained by linear transformation of the text semantic feature vector, and its dimension is d. q K represents the key vector obtained by linear transformation of audio and video composite features, with dimension d. k , and d k Usually with d q Equal; V represents the value vector obtained by linear transformation of audio and video composite features, with dimension d. v .

[0071] QK T This indicates that the attention score matrix is ​​obtained by performing a dot product operation on the transpose of the query vector and the key vector; It is the square root of the key vector dimension, used to scale the dot product result and prevent gradient saturation; Softmax is a flexible maximum function that converts the attention score into a probability distribution. The final result V is weighted and summed according to the attention weights to obtain the fused features.

[0072] The training details for the cross-attention module are as follows: Training data construction: A large-scale multimodal dataset containing text-audio-video pairings, such as MSR-VTT or ActivityNetCaptions, is used. Each sample contains a video segment, the corresponding audio, and a descriptive text, annotating the correspondence between keywords in the text and specific time segments in the audio and video.

[0073] Loss function design: A triplet loss function is adopted. For a query text, the correctly matching audio / video clips are used as positive sample pairs, and the randomly selected non-matching audio / video clips are used as negative sample pairs. The loss function encourages that the cosine similarity between positive sample pairs is higher than the similarity between negative sample pairs, and the difference is greater than a preset interval threshold.

[0074] Training objective: Minimize the triplet loss to align the text query vector with the corresponding audio / video fusion feature vector in the semantic space. During training, the linear transformation layer W... Q W K W VThe weights are continuously optimized, enabling the attention mechanism to accurately locate audio and video regions related to the text.

[0075] Step S402: Temporal correlation and dynamic weight allocation; For the time axis of the audio and video streams, the system calculates the correlation scores of visual modal features and acoustic modal features at each time slice. To improve the robustness of the system in complex acoustic environments, a signal-to-noise ratio sensing module is introduced.

[0076] This module monitors the signal-to-noise ratio (SNR) of the audio stream in real time. When the SNR falls below a third preset threshold, such as 15 dB, the system determines that the current audio environment is poor and automatically increases the weight ratio of video stream features in the fusion process. Specifically, a dynamic weighting coefficient β is set, with a value ranging from a preset range, such as 0.6 to 0.9.

[0077] The value of β is inversely proportional to the signal-to-noise ratio (SNR): the lower the SNR, the larger the β value, meaning the higher the weight of the visual features. The fused features are represented as follows: ; Wherein: F fused F represents the fused feature vector; visual F represents the visual feature vector extracted from the video frame; audio This represents the acoustic feature vector extracted from the audio stream; β is the dynamic weighting coefficient, obtained by linear interpolation within a preset range based on the real-time signal-to-noise ratio.

[0078] This dynamic weighting strategy ensures that when speech is unclear, the system can rely more on visual cues for matching, such as inferring semantics through lip movements or scene changes.

[0079] Step S403: Constructing the cross-modal semantic association matrix; Arrange the cross-modal fusion features generated in step S401 in chronological order and map them onto a unified time axis. Construct a two-dimensional matrix called the cross-modal semantic association matrix, denoted as M. The number of rows in this matrix equals the number of semantic units in the query text, such as the number of tokens after word segmentation; the number of columns equals the number of time windows in the audio / video sequence, with each time window corresponding to a fixed time length, such as 100 milliseconds.

[0080] Element M in the matrix i,j This represents the fine-grained matching probability between the i-th query semantic unit and the audio / video content within the j-th time window. This probability value is calculated from the fused features through a normalization layer and ranges from 0 to 1. Based on this matrix, the system can achieve precise alignment between audio / video content and text semantics at the millisecond level.

[0081] In summary, step S4, guided by textual semantic features, utilizes a multimodal cross-attention mechanism to weightedly fuse audio and video spatiotemporal features and acoustic features, and introduces a signal-to-noise ratio-aware dynamic weight allocation strategy to adapt to complex acoustic environments. The final constructed cross-modal semantic association matrix characterizes the matching probability of each query semantic unit with audio and video content at millisecond-level precision on the time axis.

[0082] Finally, in step S5, after constructing the cross-modal semantic association matrix in step S4, the system has obtained the fine-grained matching probability between the audio / video content and the query text within each time window. The task of this step is to locate the audio / video segments most likely to contain the user's searched keywords based on this matrix, and to eliminate false matches through a two-way verification mechanism, ultimately outputting a high-confidence location result. This is specifically achieved through the following sub-steps.

[0083] Step S501: Adaptive sliding window search; Based on the cross-modal semantic association matrix M generated in step S403, the system applies a sliding window algorithm to perform a global search on the audio and video timeline. The length of the sliding window is not a fixed value, but is adaptively adjusted according to the number of characters in the query text entered by the user.

[0084] Specifically, the lower limit of the window length is preset to 0.5 seconds, and the upper limit is preset to 5 seconds. The window length is directly proportional to the number of words in the query text: the more words, the longer the window length, in order to cover the complete semantic expression. For example, a query containing 5 terms can have a window length set to about 2 seconds.

[0085] When the sliding window moves along the timeline, the overlap between adjacent windows is preset to 50%. This overlap setting can prevent omissions caused by the window boundary falling exactly where the keyword appears.

[0086] The system calculates the cumulative matching score for all time points within the coverage area of ​​each window, which is achieved by summing or averaging the element values ​​of the corresponding column regions in matrix M. The window with the highest score is identified as a candidate segment.

[0087] Step S502: Two-way verification mechanism; To eliminate false matches caused by accidental similarity or noise, the system performs two-way verification on the candidate segments selected in step S501. The verification process is divided into two directions: forward matching and backward consistency check.

[0088] Forward matching verification: the direction from text to audio / video. The system extracts the original speech data corresponding to the candidate segments and transcribes it into text using a pre-trained automatic speech recognition model. Simultaneously, the system performs object detection on key video frames in the segment, identifying visual labels such as cars and faces. The extracted speech-to-text and visual labels together constitute the semantic content description of the segment.

[0089] Backward consistency check: from audio / video to text. The system uses a pre-trained cross-language similarity model to calculate the cosine similarity between the extracted transcribed text, visual tags, and the user's original query request. This similarity value is denoted as the backward score S. backward .

[0090] At the same time, a forward score S will also be generated during the forward matching verification. forward The score can be directly taken from the cumulative matching score of the candidate segment in step S501.

[0091] The training details for the cross-lingual similarity model are as follows: Training data construction: Collect large-scale cross-lingual text pairs, such as English-Chinese parallel corpora. Each sample contains sentences from two different languages, and their semantic similarity is labeled, such as continuous values ​​between 0 and 1 or binary labels. Simultaneously, image-text pairing data can also be used to enable the model to learn the semantic association between visual labels and text.

[0092] Loss function design: Mean squared error loss can be used for regression tasks, or binary cross-entropy loss can be used for classification tasks. For direct prediction of cosine similarity, cosine embedding loss can be used, with the following form: ; Where u and v are vector representations of the two input sentences, respectively. This is the cosine similarity function.

[0093] Training objective: To minimize the loss function so that the model can accurately calculate the semantic similarity between any two cross-lingual texts or between text and visual labels. After training, the model outputs a similarity score between 0 and 1.

[0094] Step S503: Confidence Score and Result Output; The system calculates the weighted average of the forward and backward scores as the final confidence score for the candidate segment. The calculation formula is as follows: ; Where: Score represents the final confidence score of the candidate segment, ranging from 0 to 1; α represents the preset weighting coefficient, used to balance the importance of forward and backward scores, typically ranging from 0.5 to 0.7, for example, 0.6; S forward This represents the forward matching score, i.e., the cumulative matching score of the candidate segment in step S501; S backward This represents the backward consistency test score, calculated by the cross-language similarity model.

[0095] When the final score is higher than the second preset threshold, such as 0.85, the system determines that the candidate segment is a successful match. Subsequently, the system outputs the time stamp interval with a predetermined precision corresponding to the segment, including the start and end times, and also outputs a visual label for the matching result, such as highlighting the corresponding area on the video screen or generating a text prompt.

[0096] In summary, step S5, based on the cross-modal semantic association matrix, employs an adaptive sliding window search to locate candidate segments. Then, through a bidirectional verification mechanism of forward matching and backward consistency checking, false matches are effectively eliminated. The final output timestamp interval error can be controlled within a preset range, such as within 100 milliseconds, achieving millisecond-level positioning accuracy.

[0097] On the other hand, the cross-language audio and video keyword matching system disclosed in this application includes: The preprocessing module is used to preprocess the input cross-language audio and video raw data to obtain standardized video frame sequences, clean human voice audio streams, and timestamped text metadata. The semantic representation module is used to map user-input keywords to a cross-language shared semantic representation space using a pre-trained model based on a transformer architecture, thereby obtaining text semantic feature vectors. The model uses semantically identical sentence pairs and unrelated sentence pairs from a multilingual parallel corpus as samples, and trains by contrastive learning of the objective function to make the vector distance between semantically identical cross-language sentence pairs in the space less than a first preset threshold. The feature extraction module is used to extract spatiotemporal composite features of video based on standardized video frame sequences and to extract audio acoustic fingerprint features based on clean human voice audio streams. The cross-modal fusion and matching module is used to generate fusion features by using text semantic feature vectors as query vectors and concatenating video spatiotemporal composite features and audio acoustic fingerprint features as key and value vectors, and performing cross-attention calculations. The fusion weights are adjusted through nonlinear compensation based on the slope of the real-time signal-to-noise ratio change and the motion saliency of video keyframes. The localization and verification module is used to construct a cross-modal semantic association matrix based on fused features. During the construction, a temporal causal mask is introduced, and sparsity constraints are applied by combining the topological features of cross-language grammatical structures. Based on the constrained matrix, candidate segments are located through a sliding window, and bidirectional verification is performed to generate matching results. The bidirectional verification includes the introduction of cross-language knowledge graphs for common sense logic verification in the backward consistency test.

[0098] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects.

[0099] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A cross-language audio and video keyword matching method, characterized in that, Includes the following steps: The input cross-language audio and video raw data is preprocessed to obtain standardized video frame sequences, clean human voice audio streams, and time-stamped text metadata; Using a pre-trained model based on a transformer architecture, the keywords input by the user are mapped to a cross-language shared semantic representation space to obtain text semantic feature vectors. The model uses semantically identical sentence pairs and unrelated sentence pairs from a multilingual parallel corpus as samples, and is trained by a contrastive learning objective function to make the vector distance between semantically identical cross-language sentence pairs in the space less than a first preset threshold. Video spatiotemporal composite features are extracted based on the standardized video frame sequence, and audio acoustic fingerprint features are extracted based on the pure human voice audio stream. Using the text semantic feature vector as the query vector, and the concatenation of the video spatiotemporal composite feature and the audio acoustic fingerprint feature as the key vector and value vector, cross-attention calculation is performed to generate fusion features; wherein, based on the real-time signal-to-noise ratio change slope and the motion saliency of video keyframes, the fusion weights are adjusted through nonlinear compensation. A cross-modal semantic association matrix is ​​constructed based on the fusion features. During construction, a temporal causal mask is introduced, and sparsity constraints are applied by combining the topological features of cross-language grammatical structures. Based on the constrained matrix, candidate segments are located through a sliding window, and bidirectional verification is performed to generate matching results. The bidirectional verification includes the introduction of cross-language knowledge graphs for common sense logic verification in the backward consistency test.

2. The cross-language audio and video keyword matching method according to claim 1, characterized in that, The training of a pre-trained model includes: Positive sample sentence pairs are constructed by extracting semantically identical source language sentences and target language sentences from parallel corpora of multiple languages, and semantically unrelated sentence pairs are used as negative samples. The contrastive learning objective function aims to minimize the contrastive loss value, thereby maximizing the cosine similarity of positive sample sentence pairs and minimizing the cosine similarity of negative sample sentence pairs. During training, a Euclidean distance constraint is also introduced to ensure that the Euclidean distance between semantically similar cross-linguistic words in the vector space is less than the first preset threshold.

3. The cross-language audio and video keyword matching method according to claim 2, characterized in that, Mapping user-input keywords to a cross-linguistic shared semantic representation space includes: A context-aware operator is introduced. This operator uses positional encoding information and the context vector output by the masked language model to perform weighted fusion with the static word vector of the keyword based on the semantic environment of the keyword in the query statement, so as to dynamically adjust the vector position of the keyword in the cross-language shared semantic representation space.

4. The cross-language audio and video keyword matching method according to claim 1, characterized in that, Extracting spatiotemporal composite features of video based on the standardized video frame sequence includes: The standardized video frame sequence is processed using a three-dimensional convolutional neural network, wherein the convolution kernel of the three-dimensional convolutional neural network performs convolution operations synchronously in the temporal and spatial dimensions. The output of the three-dimensional convolutional neural network is fed into a long short-term memory network or a gated recurrent unit to capture scene context, object category and action information along the time axis; A fixed-dimensional spatiotemporal composite feature vector for video is obtained by compression through a global average pooling layer.

5. The cross-language audio and video keyword matching method according to claim 1, characterized in that, Extracting audio acoustic fingerprint features based on the pure human voice audio stream includes: The pure human voice audio stream is subjected to frame segmentation and windowing processing; The power spectrum is obtained by performing a fast Fourier transform on the framed audio signal, and the power spectrum is then passed through a set of Mel filter banks to obtain the logarithmic energy distribution of each frequency band. The discrete cosine transform is applied to extract the Mel frequency cepstral coefficients, which are used as the audio acoustic fingerprint features.

6. The cross-language audio and video keyword matching method according to claim 1, characterized in that, Performing cross-attention calculations includes: The text semantic feature vector is passed through a fully connected linear transformation layer to obtain the query vector; The spliced ​​video spatiotemporal composite features and the audio acoustic fingerprint features are respectively passed through two independent linear transformation layers to obtain key vectors and value vectors; Calculate the scaled dot product of the query vector and the key vector, and obtain the attention weight map by normalization using a flexible maximum function; The attention weight map is applied to the value vector to generate the fused feature.

7. The cross-language audio and video keyword matching method according to claim 1, characterized in that, Based on the slope of the real-time signal-to-noise ratio change and the motion saliency of video keyframes, the fusion weights are adjusted through nonlinear compensation, including: When the slope of the signal-to-noise ratio indicates that the noise intensity continues to increase within a preset time window, and the motion significance of the current video keyframe exceeds a preset motion threshold, the weight of the video spatiotemporal composite features during fusion is further increased through a nonlinear compensation function.

8. The cross-language audio and video keyword matching method according to claim 1, characterized in that, Performing two-way verification on the candidate fragments includes: Forward matching verification: Extract the original speech data corresponding to the candidate segment, transcribe it into text through an automatic speech recognition model, and perform object detection on key video frames to identify visual tags, thereby constructing a semantic content description of the segment; Backward consistency test: Using a pre-trained cross-language similarity model, the cosine similarity between the semantic content description and the user's original query request is calculated to obtain the backward score; and a cross-language knowledge graph is introduced to perform common sense logic verification on the visual tags and the keywords of the user query, and the backward score is corrected based on the verification results. The cumulative matching score of the candidate fragment is calculated as the forward score. The forward score and the corrected backward score are weighted and averaged to obtain the final confidence score.

9. The cross-language audio and video keyword matching method according to claim 1, characterized in that, Locating candidate segments using a sliding window includes: An adaptive sliding window is used to search the cross-modal semantic association matrix after the sparsification constraint. The length of the adaptive sliding window is dynamically adjusted according to the number of characters in the query text, and adjacent windows have a preset overlap rate. Calculate the cumulative matching score within the coverage area of ​​each window, and select the window with the highest score as the candidate segment.

10. A cross-language audio and video keyword matching system, characterized in that, The system for performing the method as described in any one of claims 1 to 9 includes: The preprocessing module is used to preprocess the input cross-language audio and video raw data to obtain standardized video frame sequences, clean human voice audio streams, and timestamped text metadata. The semantic representation module is used to map user-input keywords to a cross-language shared semantic representation space using a pre-trained model based on a transformer architecture, thereby obtaining text semantic feature vectors. The model uses semantically identical sentence pairs and unrelated sentence pairs from a multilingual parallel corpus as samples, and is trained by a contrastive learning objective function to make the vector distance between semantically identical cross-language sentence pairs in the space less than a first preset threshold. The feature extraction module is used to extract video spatiotemporal composite features based on the standardized video frame sequence and to extract audio acoustic fingerprint features based on the pure human voice audio stream. The cross-modal fusion and matching module is used to generate fusion features by performing cross-attention calculation using the text semantic feature vector as the query vector and the concatenation of the video spatiotemporal composite features and audio acoustic fingerprint features as the key vector and value vector; wherein, the fusion weights are adjusted by nonlinear compensation based on the slope of the real-time signal-to-noise ratio change and the motion saliency of the video keyframes. The localization and verification module is used to construct a cross-modal semantic association matrix based on the fusion features. During construction, a temporal causal mask is introduced, and sparsity constraints are applied by combining the topological features of cross-language grammatical structures. Based on the constrained matrix, candidate segments are located through a sliding window, and bidirectional verification is performed to generate matching results. The bidirectional verification includes the introduction of a cross-language knowledge graph for common sense logic verification in the backward consistency test.

Images