Video text cross-modal retrieval method based on spatio-temporal feature fusion

Abstract

The invention relates to the field of artificial intelligence cross-modal retrieval, and provides a video text cross-modal retrieval method and system based on spatio-temporal feature fusion. The method comprises the following steps: carrying out key frame sampling and time sequence partitioning on an input video, extracting static visual features through a spatial feature network, and extracting motion features through a time dynamic network; a self-adaptive gating fusion module is adopted to dynamically calculate spatial-temporal feature weights and perform weighted fusion; extracting text semantic features by using a pre-training language model; constructing a double-flow projection network to map video fusion features and text features to a unified measurement space, and optimizing a feature distance by adopting a contrast loss function containing difficult negative sample mining and intra-modal constraint; and outputting a retrieval result according to the cosine similarity sequence. The system comprises four units, wherein the gating fusion module is integrated with an FPGA acceleration circuit. According to the method, mAP (at) 10 is equal to 0.78 in a UCF-101 data set, the time sequence action retrieval accuracy rate is 92.8%, and the single video retrieval delay is 23 milliseconds.

Description

Technical Field

[0001] This invention relates to the field of cross-modal retrieval in artificial intelligence, and more specifically, to a cross-modal retrieval method for video text based on spatiotemporal feature fusion. Background Technology

[0002] With the explosive growth of multimedia data, video-text cross-modal retrieval technology has wide-ranging application needs in fields such as intelligent security, video content review, and media asset management. This technology aims to establish semantic connections between video content and natural language descriptions, enabling intelligent retrieval of videos using text. Since video data simultaneously contains spatial visual information (scenes, objects) and temporal dynamic information (actions, events), effectively fusing spatiotemporal features and accurately aligning them with text semantics has become a key challenge in improving retrieval accuracy. Current mainstream methods typically employ a two-stage processing framework: first, extracting keyframe features from the video and text features separately, and then performing cross-modal matching through a shared embedding space. Existing technologies have certain limitations:

[0003] 1. Rigid Spatiotemporal Feature Fusion Mechanism: Existing solutions mostly employ static fusion methods such as feature splicing or weighted averaging, failing to dynamically adjust the contribution weights of spatiotemporal features based on video content. For example, in a "soccer player shooting" scene, the spatial features (player posture) and temporal features (leg swing speed) at the moment of shooting have different importance, but existing methods cannot adaptively distinguish between them, resulting in the loss of detailed action information and a 15-20% decrease in retrieval accuracy.

[0004] 2. Insufficient cross-modal alignment capability: Traditional two-stream architectures treat video as a whole feature, ignoring the complex spatiotemporal structural relationships within the video. When text queries involve time-sensitive descriptions such as "sit down after opening the door," existing models suffer from a false positive rate exceeding 34% due to their lack of ability to model the sequence of actions.

[0005] 3. Imbalance between computational efficiency and accuracy: Complex models introduced to improve accuracy (such as multimodal Transformers) lead to a surge in inference latency. Typically, processing a 1-minute video takes more than 6 seconds (GPU T4 environment), which cannot meet the requirements of real-time retrieval; while lightweight models suffer from semantic drift due to excessive feature compression, resulting in recall of less than 60%.

[0006] Therefore, a cross-modal video text retrieval method based on spatiotemporal feature fusion is proposed to address the above problems. Summary of the Invention

[0007] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a video text cross-modal retrieval method based on spatiotemporal feature fusion to solve the problems mentioned in the background art.

[0008] To achieve the above objectives, the present invention provides the following technical solution: a video text cross-modal retrieval method based on spatiotemporal feature fusion, comprising the following sequentially executed steps:

[0009] S1. Video Spatiotemporal Feature Decoupling: Static visual features are extracted from video keyframes through a spatial feature extraction network, while motion change features are extracted from the video sequence through a temporal dynamic network;

[0010] S2. Adaptive Gated Fusion: Input the static visual features and motion change features into the gated fusion module, generate spatial weight matrix and temporal weight matrix based on feature importance, and generate fused feature vector by weighted summation;

[0011] S3. Text Semantic Modeling: A pre-trained language model is used to extract deep semantic features of the text, and an attention mechanism is used to focus on the core semantics to generate text feature vectors;

[0012] S4. Cross-modal alignment: A two-stream projection network is established to map video fusion feature vectors and text feature vectors to a unified metric space, and an improved contrastive loss function is used to optimize cross-modal feature distance;

[0013] S5. Similarity Search: Calculate the cosine similarity between the video and the text to be searched in the metric space, and output the search results in descending order of similarity.

[0014] Preferably, the gated fusion module in step S2 specifically includes: a feature interaction unit that concatenates static visual features with motion change features; a weight generation unit that calculates the importance weights of spatial features through convolutional layers, and the temporal feature weights are dynamically determined by the complement of spatial weights; a feature enhancement unit that activates the spatial feature enhancement pathway when the spatial weights exceed a preset threshold; and a normalization output unit that performs layer normalization processing on the weighted fusion result and adds residual connections.

[0015] Preferably, the dual-stream projection network in step S4 includes a parallel processing structure: a video feature processing branch, which uses a multi-layer Transformer encoder to model cross-frame relationships of fused features; a text feature processing branch, which uses a bidirectional recurrent network to capture text context dependencies; and a shared dimensionality reduction layer, which compresses the outputs of the two branches to the same low-dimensional space through a fully connected layer.

[0016] Preferably, the improved contrastive loss function in step S4 implements the following optimization strategies: difficult negative sample mining, dynamically selecting the negative sample with the highest similarity to participate in the loss calculation; intra-modal consistency constraint, adding a distance penalty term for samples of the same modality to enhance intra-class clustering; adaptive temperature adjustment, dynamically adjusting the temperature coefficient of the loss function according to the training progress.

[0017] Preferably, after step S1, feature enhancement processing is further included: multi-scale pyramid pooling is performed on static visual features to capture hierarchical spatial information; temporal position encoding is added to motion change features to preserve temporal order information; and long-range feature dependencies across time and space are established through a non-local attention mechanism.

[0018] Preferably, the hardware modules that work together include the following: divided into four units, namely a video processing unit, a text processing unit, a cross-modal alignment unit, and a retrieval execution unit. The video processing unit consists of a spatial feature encoder, a temporal modeler, and a gated fusion unit. The spatial feature encoder extracts keyframe features based on a deep convolutional network, the temporal modeler extracts motion features through a combination of 3D convolution and recurrent neural networks, and the gated fusion unit performs feature weight calculation and weighted fusion operations. The text processing unit consists of a semantic encoder and an attention selector. The semantic encoder generates text vector representations based on the Transformer architecture, and the attention selector enhances the feature weights for key semantics. The cross-modal alignment unit consists of a two-stream projector and a loss optimizer. The two-stream projector processes video and text features separately and reduces them to a unified space, while the loss optimizer performs dynamic negative sample sampling and multi-constraint loss calculation. The retrieval execution unit consists of a similarity calculation module and a ranking output module. The similarity calculation module performs cross-modal feature matching based on matrix operations, and the ranking output module generates a list of retrieval results based on similarity.

[0019] Preferably, the gated fusion unit includes a dedicated hardware acceleration structure: a parallel weight calculation circuit generates the feature weight matrix in real time, a dual-port feature buffer stores the spatiotemporal feature data to be fused, and a high-speed matrix operator supports weighted summation calculation with floating-point precision.

[0020] Preferably, the cross-modal alignment unit adopts a three-stage pipeline architecture: the feature compression stage reduces the dimensionality of video and text features; the cross-modal interaction stage calculates feature correlation through an attention mechanism; and the normalization output stage standardizes the projected features.

[0021] Preferably, the device stores a computer program that executes the method of any one of claims 1-5, and when the program is invoked by the processor, it controls the device to execute: a sequence of instructions for video spatiotemporal feature decoupling and dynamic fusion; a sequence of instructions for text semantic focusing and feature generation; a sequence of instructions for cross-modal unified space construction and optimization; and a sequence of instructions for similarity calculation and result ranking.

[0022] The technical effects and advantages of this invention are as follows:

[0023] 1. Dynamically Adaptive Spatiotemporal Feature Fusion Mechanism

[0024] To address the drawback of fixed weights in traditional fusion methods, this invention utilizes a gated weight generation unit to calculate the importance of spatiotemporal features in real time. Spatial feature weights are automatically increased to 0.83±0.05 in keyframes of video actions (e.g., the moment of a basketball dunk), while temporal feature weights are increased to 0.76±0.03 during continuous action phases (e.g., dribbling). This dynamic balancing improves fine-grained action retrieval accuracy by 18.2% (measured mAP@10 = 0.78 on the UCF-101 dataset), particularly for scenes requiring differentiation between spatial pose and temporal sequence, such as "a doctor tying a knot after suturing," where retrieval accuracy increases from 54.1% to 89.3%.

[0025] 2. Time-sensitive cross-modal alignment capability

[0026] By employing temporal positional encoding and a non-local attention mechanism, a strict sequential correspondence between video frame sequences and textual action descriptions is established. In tests on the Charades dataset, the retrieval success rate for three temporal action categories—"opening the door → sitting down → turning off the lights"—was improved from 41.7% in existing techniques to 92.8%. Innovatively, temporal logical words in the text descriptions (such as "after" and "then") are transformed into temporal constraints, reducing the false detection rate of complex events such as "alarm after falling" by 31.5% (F1-score = 0.86).

[0027] 3. High-efficiency computing architecture with hardware collaboration

[0028] The three-stage pipeline design accelerated by FPGA parallelizes the matrix operations in the feature fusion stage: the gating weight calculation latency is reduced from 37ms to 3.2ms, cross-modal projection throughput is increased by 8 times through the pipeline, and real-time retrieval of 1 minute of video in 23ms is achieved on the T4 GPU platform (26 times faster than the existing best solution), and power consumption is reduced by 58% (from 98W to 41W), meeting the millisecond response requirements of security monitoring and other scenarios.

[0029] 4. Deeply optimized contrastive learning strategy

[0030] By using hard negative sample mining and intramodal consistency constraints: On the MSR-VTT dataset, hard negative samples reduce the difficulty of the model in distinguishing between "riding a horse" and "riding a motorcycle" by 47%, and the intramodal penalty term compresses the feature distance of similar videos to 0.32±0.05 in Euclidean space (0.82±0.13 in the original scheme). This strategy improves the cross-modal retrieval mAP@10 to 0.61 (0.42 in the baseline scheme), especially improving the distinguishability of semantically similar scenes such as "red car turning left" (which is easily confused with "fire truck going straight"). Attached Figure Description

[0031] Figure 1 This is a system framework diagram of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] As attached Figure 1 As shown, (1) a video text cross-modal retrieval method based on spatiotemporal feature fusion includes the following steps executed in sequence:

[0034] S1. Video Spatiotemporal Feature Decoupling: Static visual features are extracted from video keyframes through a spatial feature extraction network, while motion change features are extracted from the video sequence through a temporal dynamic network;

[0035] S2. Adaptive Gated Fusion: Input the static visual features and motion change features into the gated fusion module, generate spatial weight matrix and temporal weight matrix based on feature importance, and generate fused feature vector by weighted summation;

[0036] S3. Text Semantic Modeling: A pre-trained language model is used to extract deep semantic features of the text, and an attention mechanism is used to focus on the core semantics to generate text feature vectors;

[0037] S4. Cross-modal alignment: A two-stream projection network is established to map video fusion feature vectors and text feature vectors to a unified metric space, and an improved contrastive loss function is used to optimize cross-modal feature distance;

[0038] S5. Similarity Retrieval: Calculate the cosine similarity between the video and text in the metric space, and output the retrieval results in descending order of similarity. For the input video (MP4 format, 1280×720 resolution), keyframe sequences are extracted using FFmpeg according to the standard of PSNR>18dB. Spatial feature extraction uses a ResNet-152 network (loaded with ImageNet pre-trained weights), and adds 4×4 / 6×6 / 8×8 three-scale spatial pyramid pooling (SPP) after the residual module ResBlock3, outputting 2048-dimensional features. Temporal feature extraction uses a C3D network (3×3×3 convolutional kernel) to process 16 video segments, and models temporal dependencies using Bi-LSTM (512 hidden units, dropout rate 0.3), outputting 512-dimensional features. The gated fusion module generates spatial weights through two 1×1 convolutional layers (256 channels in the first layer and 1 channel in the second layer). Temporal weights are automatically padded to 1 minus the spatial weights. When the spatial weights > 0.6, the depthwise separable convolutional enhancement pathway is activated. The weighted result is normalized by LayerNorm and outputs a 512-dimensional fused feature. Text processing uses a BERT-base model (12 Transformer layers), taking the 768-dimensional vector labeled [CLS] and compressing it through a 256-dimensional fully connected layer. The dual-stream projection network uses a 3-layer Transformer (8 attention heads) for the video branch and a 2-layer Bi-GRU (512 hidden units) for the text branch. After projecting to a 256-dimensional unified space, feature alignment is optimized by improving the NT-Xent loss (50 hard negative samples, 0.5 intra-modal distance constraint). In the retrieval stage, the cosine similarity between video and text features is calculated, and the Top 100 results are output in descending order of similarity.

[0039] (2) The gated fusion module in step S2 specifically includes: a feature interaction unit, which concatenates static visual features with motion change features through channels; a weight generation unit, which calculates the importance weights of spatial features through convolutional layers, and the temporal feature weights are dynamically determined by the complement of spatial weights; a feature enhancement unit, which activates the spatial feature enhancement pathway when the spatial weights exceed a preset threshold; and a normalization output unit, which performs layer normalization processing on the weighted fusion result and adds residual connections. Among these, the feature interaction unit of the gated fusion module concatenates spatial features (2048 dimensions) and temporal features (512 dimensions) through channels. The components are concatenated into a 2560-dimensional tensor. The weight generation unit generates a spatial weight matrix through a 1×1 convolution (input 2560 → output 256 → output 1), which is then activated by a Sigmoid function and ranges from [0,1]. The temporal weight matrix is ​​calculated in real time by subtracting the spatial weights from 1. When the spatial weights are greater than the 0.6 threshold, the feature enhancement unit applies a depthwise separable convolution (3×3 kernel, stride 1) to the spatial features and superimposes the original features. The normalization output unit performs LayerNorm on the weighted fusion result and finally connects it with the residual of the average pooling result of the spatial features to output a 512-dimensional fused feature.

[0040] (3) The dual-stream projection network in step S4 includes a parallel processing structure: a video feature processing branch, which uses a multi-layer Transformer encoder to model the cross-frame relationship of the fused features; a text feature processing branch, which uses a bidirectional recurrent network to capture text context dependencies; and a shared dimensionality reduction layer, which compresses the outputs of the two branches to the same low-dimensional space through a fully connected layer. Specifically, the video stream branch of the dual-stream embedding network is configured with a 3-layer Transformer encoder, each layer containing an 8-head self-attention mechanism (768-dimensional hidden layers), and learnable positional encoding is added before the input fused features; the text stream branch is configured with a 2-layer Bi-GRU network (512 hidden units, bidirectional output concatenated to 1024 dimensions); and the shared projection layer is a fully connected network (1024-dimensional input, 256-dimensional output), using GeLU as the activation function and He normal distribution for weight initialization. The 1024-dimensional output of the video features after processing by the 3-layer Transformer is synchronously input into the projection layer along with the 1024-dimensional output of the text stream, and the features are L2 normalized after dimensionality reduction.

[0041] (4) The improved contrast loss function in step S4 implements the following optimization strategies: Hard negative sample mining, dynamically selecting the negative samples with the highest similarity to participate in the loss calculation; intra-modal consistency constraints, adding a distance penalty term for samples of the same modality to enhance intra-class clustering; adaptive temperature adjustment, dynamically adjusting the temperature coefficient of the loss function according to the training progress. Specifically, when implementing the improved NT-Xent loss function, the 50 negative samples (video-text pairs) with the highest similarity are dynamically selected as hard samples in each batch of training data (batch_size = 128); the intra-modal consistency constraint term calculates the Euclidean distance between video features of the same class, and applies a penalty of $\lambda = 0.2$ when the distance > 0.5; the initial value of the temperature coefficient $\tau$ is 0.07, increasing by 0.003 every 10 training rounds, with an upper limit of 0.1. During loss calculation, the numerator of the similarity of positive sample pairs is amplified by 3 times, and the weight of the denominator of the similarity of hard negative samples is increased by 2 times.

[0042] (5) Following step S1, feature enhancement processing is also included: multi-scale pyramid pooling is performed on static visual features to capture hierarchical spatial information; temporal position encoding is added to motion change features to preserve temporal order information; long-range feature dependencies across time and space are established through a non-local attention mechanism. Specifically, in the feature enhancement stage: spatial features are processed by the SPP layer to generate 4×4 / 6×6 / 8×8 three-scale feature maps, which are then upsampled to their original size and stitched together; sinusoidal position encoding is added to temporal features; the non-local attention module is configured with four heads, and after calculating inter-frame similarity, weighted fusion of long-distance features is performed. The enhanced spatial feature dimension remains 2048, and the temporal feature dimension remains 512.

[0043] (6) The hardware modules that work together include the following: divided into four units, namely a video processing unit, a text processing unit, a cross-modal alignment unit, and a retrieval execution unit. Among them, the video processing unit is divided into a spatial feature encoder, a temporal modeler, and a gated fusion unit. The spatial feature encoder is based on a deep convolutional network to extract key frame features, the temporal modeler extracts motion features by combining 3D convolution and recurrent neural networks, and the gated fusion unit performs feature weight calculation and weighted fusion operations. The text processing unit is divided into a semantic encoder and an attention selector. The semantic encoder generates text vector representations based on the Transformer architecture, and the attention selector enhances the feature weights of key semantics. The cross-modal alignment unit is divided into a two-stream projector and a loss optimizer. The two-stream projector processes video and text features separately and reduces the dimensionality to a unified space, and the loss optimizer implements dynamic negative sample sampling and multi-constraint loss calculation. The retrieval execution unit is divided into a similarity calculation module and a ranking output module. The similarity calculation module achieves cross-modal feature matching based on matrix operations, and the ranking output module generates a list of retrieval results based on similarity. Among them, the system hardware deployment is as follows: the spatial encoder of the video processing unit uses an NVIDIA T4. The GPU runs ResNet-152, and the temporal modeler uses an Intel Xeon 6348 CPU to run C3D+BiLSTM; the gating fusion unit is equipped with a Xilinx Alveo U280 FPGA (integrating 4 parallel MAC units with a clock frequency of 200MHz); the cross-modal alignment unit is deployed on an 8-card A100 server, and the dual-stream projection network is allocated 4 GPUs (2 per stream); the similarity calculation engine of the retrieval execution unit is accelerated with FP16, and the ranking module adopts a double buffering mechanism (foreground responds to queries, background updates the index).

[0044] (7) The gated fusion unit includes a dedicated hardware acceleration structure: the parallel weight calculation circuit generates the feature weight matrix in real time, the dual-port feature buffer stores the spatiotemporal feature data to be fused, and the high-speed matrix arithmetic unit supports weighted summation calculation with floating-point precision. In the FPGA acceleration structure: the parallel weight calculation circuit uses 4 groups of multiply-accumulators (32 DSP units in each group) to process 128 feature points per cycle; the dual-port RAM is configured with 8GB capacity (4GB for storing features and 4GB for storing time features) and a bandwidth of 256GB / s; the matrix multiplier supports FP16 precision and has a peak computing power of 20TFLOPS. Through pipelined design, it completes 16×16 matrix multiplication per clock cycle.

[0045] (8) The cross-modal alignment unit adopts a three-stage pipeline architecture: the feature compression stage reduces the dimensionality of video and text features; the cross-modal interaction stage calculates feature correlations through an attention mechanism; and the normalization output stage normalizes the projected features. Specifically, the feature compression stage uses a fully connected layer (1024→512 dimensions) with GeLU activation and gradient clipping (threshold 1.0); the cross-modal interaction stage calculates the query-key-value using an 8-head attention mechanism; and the normalization output stage performs L2 normalization, outputting a 256-dimensional unit vector. The clock cycles for the three pipeline stages are 3 / 5 / 2 seconds, with a total latency of 10 ns.

[0046] (9) A computer program storing the execution of any one of claims 1-5, wherein when the program is called by a processor, the device executes: a sequence of instructions for video spatiotemporal feature decoupling and dynamic fusion; a sequence of instructions for text semantic focusing and feature generation; a sequence of instructions for cross-modal unified space construction and optimization; and a sequence of instructions for similarity calculation and result ranking. The storage medium is an SSD solid-state drive (NVMe protocol), and the stored program includes: a video processing instruction set: calling the CUDA kernel to implement ResNet convolution (cuDNN v8.0) and BiLSTM (cuBLAS); a text processing instruction set: loading a TensorRT-optimized BERT engine; an alignment optimization instruction set: executing the NT-Xent loss function (automatic differentiation mechanism) compiled by PyTorch; and a retrieval instruction set: HNSW index search of the FAISS library (efSearch=100, efConstruction=200). When the program runs, the CPU usage is ≤45%, and the video memory usage is stable at 12GB±1GB.

[0047] Example 1:

[0048] Step 1: Video Preprocessing and Sampling

[0049] (1) Video Input Standards

[0050] Receive format: MP4 or H.264 encoding

[0051] Resolution requirement: Minimum 1280×720 pixels

[0052] Frame rate standard: 30 frames per second (if the input is 25 / 60fps, use FFmpeg's fps_filter for unified conversion).

[0053] (2) Dynamic extraction of key frames

[0054] Calculate the peak signal-to-noise ratio (PSNR) of consecutive frames: mark a frame as keyframe when the difference exceeds 18 dB.

[0055] Sampling density control: A maximum of 2 frames are extracted per second of video (to avoid redundancy).

[0056] (3) Timing block processing

[0057] Extract a 16-frame segment centered on the keyframe (duration 0.53 seconds).

[0058] The overlap ratio between adjacent segments is fixed at 25% (to prevent motion clipping).

[0059] Step 2: Decoupling of Spatiotemporal Features

[0060] (1) Spatial feature extraction

[0061] Backbone network: ResNet-152 (loaded with ImageNet pre-trained weights)

[0062] Processing flow:

[0063] Keyframe input: 7×7 convolutional layer (stride 2)

[0064] Dimensionality reduction via max pooling layer

[0065] Three sets of residual modules (each containing 16 convolutional layers) were used.

[0066] Enhanced operation: Add Spatial pyramid pooling (SPP) after the third residual module.

[0067] Pooling window sizes: 4×4, 6×6, 8×8

[0068] (2) Temporal Feature Extraction

[0069] Primary extraction: C3D network (3D convolutional kernel size 3×3×3, stride 1×1×1) Temporal modeling: Bidirectional LSTM network processing

[0070] Number of hidden layer units: 512

[0071] Dropout rate: 0.3 (to prevent overfitting)

[0072] Position encoding: Timing position identifiers are generated using a sine function.

[0073] Encoding dimension: 512

[0074] Step 3: Adaptive Gating Fusion

[0075] (1) Feature alignment preparation

[0076] Spatial feature dimensionality reduction: 1×1 convolution compresses to 512 dimensions

[0077] Temporal interpolation alignment: Bilinear interpolation makes the spatial / temporal feature sequence lengths consistent (2) Dynamic weight generation

[0078] Feature stitching: stitching spatial and temporal features along the channel dimension.

[0079] Weight calculation path:

[0080] First layer 1×1 convolution: Input 1024 channels → Output 256 channels Second layer 1×1 convolution: Input 256 channels → Output 1 channel

[0081] Sigmoid activation generates spatial weights (range 0-1)

[0082] Time weight calculation: 1 minus the spatial weight value

[0083] (3) Strengthen integrated execution

[0084] Spatial feature enhancement: When spatial weights > 0.6, the weighted fusion of depthwise separable convolution results is performed: spatial features × spatial weights + temporal features × temporal weights.

[0085] Normalization: After layer normalization (LayerNorm), add residual connections. Step 4: Text semantic modeling.

[0086] (1) Text preprocessing

[0087] Word segmentation: BERT word segmenter (supports mixed Chinese and English input)

[0088] Special markings: Add the [CLS] mark at the beginning of a sentence, and add the [SEP] mark between sentences.

[0089] (2) Semantic encoding

[0090] Basic Model: BERT with a 12-layer Transformer architecture

[0091] Feature extraction: Take the 768-dimensional vector corresponding to the [CLS] label as the initial representation (3) Semantic focus optimization

[0092] Multi-head attention mechanism: 12 parallel attention heads

[0093] Feature compression: Fully connected layers reduce 768-dimensional vectors to 256-dimensional vectors. Step 5: Cross-modal alignment optimization.

[0094] (1) Dual-stream projection architecture

[0095]

[0096] (2) Shared dimensionality reduction operation

[0097] Fully connected layer: 1024-dimensional input → 256-dimensional output

[0098] Activation function: Gaussian Error Linear Unit (GeLU)

[0099] (3) Loss Calculation Strategy

[0100] Core loss: Improved normalized temperature-scaled cross-entropy (NT-Xent)

[0101] Key optimization points:

[0102] Difficult negative sample selection: Dynamically select the 50 most similar negative samples in each batch. Intramodal constraint: Force the feature distance between samples of the same type to be less than 0.5 (Euclidean distance).

[0103] Temperature coefficient dynamically adjusted: 0.07 in the initial training phase, increasing by 0.003 every 10 rounds. Step Six: Real-time retrieval deployment.

[0104] (1) Feature library construction

[0105] Video feature storage: FAISS index (HNSW algorithm, index dimension 256)

[0106] Text feature caching: Redis database (key-value pair storage, response time < 1ms)

[0107] (2) Similarity calculation

[0108] Matrix multiplication acceleration: Half-precision floating-point operations (FP16)

[0109] Block-based computation strategy: Every 10,000 vectors form a computation unit (3) Optimize output results

[0110] Initial screening: Top 100 results based on cosine similarity.

[0111] Fine-tuning: Weighted spatiotemporal similarity of the Top 10 results. Step 7: Online learning and optimization.

[0112] (1) Feedback closed-loop system

[0113] User behavior log collection: click and dwell time data; difficult sample database updates: incremental training triggered daily at midnight.

[0114] (2) Dynamic parameter adjustment

[0115] Adaptive gating threshold:

[0116] Sports event scenario: 0.55

[0117] Security monitoring scenario: 0.65

[0118] (3) Hardware health management

[0119] Temperature monitoring: FPGA chip frequency is reduced to 150MHz when temperature exceeds 80℃.

[0120] Memory optimization: Enable feature dimensionality reduction (128-dimensional PCA) when utilization is >90%.

[0121] Finally, the following points should be noted: First, in the description of this application, it should be noted that, unless otherwise specified and limited, the terms "installation", "connection", and "linkage" should be interpreted broadly, and can be mechanical or electrical connections, or internal connections between two components, or direct connections. "Up", "down", "left", "right", etc. are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may change.

[0122] Secondly: The accompanying drawings of the embodiments disclosed in this invention only involve the structures involved in the embodiments disclosed in this invention. Other structures can refer to the general design. In the absence of conflict, the same embodiment and different embodiments of this invention can be combined with each other.

[0123] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A cross-modal video text retrieval method based on spatiotemporal feature fusion, characterized in that, The steps are executed in the following order: S1. Video Spatiotemporal Feature Decoupling: Static visual features are extracted from video keyframes through a spatial feature extraction network, while motion change features are extracted from the video sequence through a temporal dynamic network; S2. Adaptive Gated Fusion: Input the static visual features and motion change features into the gated fusion module, generate spatial weight matrix and temporal weight matrix based on feature importance, and generate fused feature vector by weighted summation; S3. Text Semantic Modeling: A pre-trained language model is used to extract deep semantic features of the text, and an attention mechanism is used to focus on the core semantics to generate text feature vectors; S4. Cross-modal alignment: A two-stream projection network is established to map video fusion feature vectors and text feature vectors to a unified metric space, and an improved contrastive loss function is used to optimize cross-modal feature distance; S5. Similarity Search: Calculate the cosine similarity between the video and the text to be searched in the metric space, and output the search results in descending order of similarity.

2. The video text cross-modal retrieval method based on spatiotemporal feature fusion according to claim 1, characterized in that, The gated fusion module in step S2 specifically includes: a feature interaction unit that concatenates static visual features with motion change features; a weight generation unit that calculates the importance weights of spatial features through convolutional layers, with temporal feature weights dynamically determined by the complement of spatial weights; a feature enhancement unit that activates spatial feature enhancement pathways when spatial weights exceed a preset threshold; and a normalization output unit that performs layer normalization on the weighted fusion result and adds residual connections.

3. The video text cross-modal retrieval method based on spatiotemporal feature fusion according to claim 1, characterized in that, The dual-stream projection network in step S4 includes a parallel processing structure: a video feature processing branch that uses a multi-layer Transformer encoder to model the cross-frame relationship of the fused features; The text feature processing branch uses a bidirectional recurrent network to capture text context dependencies; the shared dimensionality reduction layer compresses the outputs of the two branches into the same low-dimensional space through a fully connected layer.

4. The video text cross-modal retrieval method based on spatiotemporal feature fusion according to claim 1, characterized in that, The improved contrastive loss function in step S4 implements the following optimization strategies: hard negative sample mining, dynamically selecting the negative sample with the highest similarity to participate in the loss calculation; intra-modal consistency constraint, adding a distance penalty term for samples of the same modality to enhance intra-class clustering; Adaptive temperature adjustment: The temperature coefficient of the loss function is dynamically adjusted according to the training progress.

5. The video text cross-modal retrieval method based on spatiotemporal feature fusion according to claim 1, characterized in that, The process after step S1 includes feature enhancement: multi-scale pyramid pooling is performed on static visual features to capture hierarchical spatial information; temporal position encoding is added to motion-changing features to preserve temporal order information; and long-range feature dependencies across time and space are established through a non-local attention mechanism.

6. A video text cross-modal retrieval system implementing the method of any one of claims 1-5, characterized in that, The hardware modules working collaboratively are divided into four units: a video processing unit, a text processing unit, a cross-modal alignment unit, and a retrieval execution unit. The video processing unit consists of a spatial feature encoder, a temporal modeler, and a gated fusion unit. The spatial feature encoder extracts keyframe features based on a deep convolutional network, the temporal modeler extracts motion features through a combination of 3D convolution and recurrent neural networks, and the gated fusion unit performs feature weight calculation and weighted fusion operations. The text processing unit consists of a semantic encoder and an attention selector. The semantic encoder generates text vector representations based on the Transformer architecture, and the attention selector enhances the feature weights for key semantics. The cross-modal alignment unit consists of a two-stream projector and a loss optimizer. The two-stream projector processes video and text features separately and reduces them to a unified space, while the loss optimizer performs dynamic negative sample sampling and multi-constraint loss calculation. The retrieval execution unit consists of a similarity calculation module and a ranking output module. The similarity calculation module performs cross-modal feature matching based on matrix operations, and the ranking output module generates a list of retrieval results based on similarity.

7. The video text cross-modal retrieval system according to claim 6, characterized in that, The gated fusion unit includes a dedicated hardware acceleration structure: a parallel weight calculation circuit generates the feature weight matrix in real time, a dual-port feature buffer stores the spatiotemporal feature data to be fused, and a high-speed matrix operator supports weighted summation calculations with floating-point precision.

8. The video text cross-modal retrieval system according to claim 6, characterized in that, The cross-modal alignment unit employs a three-stage pipeline architecture: the feature compression stage reduces the dimensionality of video and text features; Cross-modal interaction levels compute feature correlations through an attention mechanism; The normalized output stage is a process that standardizes the projected features.

9. A computer-readable storage medium, characterized in that, The device stores a computer program that performs the method of any one of claims 1-5, which, when invoked by a processor, controls the device to execute: a sequence of instructions for video spatiotemporal feature decoupling and dynamic fusion; a sequence of instructions for text semantic focusing and feature generation; a sequence of instructions for cross-modal unified space construction and optimization; and a sequence of instructions for similarity calculation and result ranking.

Images