Multi-modal video feature extraction method and device, equipment and medium

Abstract

The invention discloses a multi-mode video feature extraction method, device and equipment and a medium, which are applied to application scenes such as urban management, medical prediction and financial prediction. The method comprises the following steps: acquiring original multi-modal data through acquisition equipment, preprocessing, and performing multi-scale feature extraction on each modal data by adopting a layered multi-modal decomposition technology to generate layered features with rich spatio-temporal information. The hierarchical features are subjected to joint sparse representation optimization by using adaptive sparse coding, a sparse threshold value and a dictionary are dynamically adjusted, and efficient compression and discriminant power improvement of the multi-modal features are realized. And fusing the optimized sparse codes into a compact multi-modal feature vector, and outputting the compact multi-modal feature vector for a downstream analysis task. The problem of insufficient feature expression caused by a fixed dictionary and static fusion in the prior art is effectively solved, and the expression ability of the multi-modal video features and the ability of adapting to dynamic scenes are remarkably improved.

Claims

1. A multimodal video feature extraction method, characterized in that: include: Obtaining original multimodal data through acquisition equipment; Preprocessing the original multimodal data; Performing multimodal decomposition on the preprocessed original multimodal data to generate hierarchical features of each modality; Performing joint sparse representation optimization on the hierarchical features through adaptive sparse coding to obtain optimized sparse coding; The optimized sparse coding is integrated to generate a compact multimodal feature vector for output to downstream tasks.

2. The multimodal video feature extraction method according to claim 1, wherein: The step of performing joint sparse representation optimization on the hierarchical features by adaptive sparse coding to obtain optimized sparse coding includes: Dynamically adjust the sparse threshold based on scene complexity, and generate sparse codes for each modality according to the hierarchical features and the sparse threshold through online dictionary learning; The optimized sparse coding is obtained by optimizing the sparse coding by transferring saliency information between the sparse codings of each modality through a cross-modal attention mechanism.

3. The multimodal video feature extraction method according to claim 2, wherein: The step of dynamically adjusting the sparse threshold based on scene complexity and generating sparse codes of each modality according to the hierarchical features and the sparse threshold through online dictionary learning includes: By analyzing the number of moving objects, audio energy change rate and text keyword density in the hierarchical features, a scene complexity value is comprehensively calculated; adjusting the sparse threshold according to the scene complexity value; The hierarchical features are used as input, and a sparse code is generated by iteratively optimizing an objective function according to the sparse threshold.

4. The multimodal video feature extraction method according to claim 3, wherein: The steps of optimizing the sparse coding by transferring saliency information between the sparse codings of each modality through a cross-modal attention mechanism to obtain the optimized sparse coding include: Projecting the sparse codes of each modality into a query vector, a key vector, and a value vector respectively; Calculating an inter-modality attention weight based on the query vector, the key vector, and the value vector; Based on the attention weight weighted fusion value vector, the optimized sparse code is generated.

5. The multimodal video feature extraction method according to claim 1, wherein: The original multimodal data includes a video stream, an audio stream, and a text stream, and the step of preprocessing the original multimodal data includes: Performing frame alignment and color normalization processing on the video stream to obtain a standardized video stream; Performing noise reduction and spectrum normalization processing on the audio stream to obtain a standardized audio stream; The text stream is subjected to word segmentation and embedding vectorization processing to obtain a standardized text stream.

6. The multimodal video feature extraction method according to claim 5, characterized in that: The hierarchical features include multi-resolution spatial features, time-frequency domain energy distribution features, and phrase-level semantic unit features. The step of performing multimodal decomposition on the pre-processed original multimodal data includes: Performing multi-scale spatial decomposition on the standardized video stream, and generating the multi-resolution spatial features through a Laplacian pyramid; Performing frequency band energy decomposition on the standardized audio stream, and generating the time-frequency domain energy distribution feature through a learnable Mel filter bank; The standardized text stream is semantically segmented, and the phrase-level semantic unit features are generated through phrase-level division of a pre-trained language model.

7. The multimodal video feature extraction method according to claim 1, wherein: The step of fusing the optimized sparse coding to generate a compact multimodal feature vector for output to a downstream task includes: Splicing the optimized sparse codes of each modality along the feature dimension to form an initial fusion feature; Performing a linear transformation on the initial fusion features through a lightweight fully connected layer to generate a compact multimodal feature vector; The multimodal feature vector is input into the downstream task module.

8. A multimodal video feature extraction device, characterized in that: Used to perform the multimodal video feature extraction method as described in any one of claims 1 to 7.

9. A computer device, characterized in that: The computer device includes a memory and a processor connected to the memory; the memory is used to store a computer program; the processor is used to run the computer program stored in the memory to perform the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 can be implemented.

Images