Audio and video content intelligent retrieval method based on multi-modal features

Abstract

The invention relates to the technical field of computers, and discloses an audio and video content intelligent retrieval method based on multi-modal features, comprising the following steps: S1, preprocessing original audio and video data, including audio noise reduction, video frame extraction and screening, and audio and video segmentation, to obtain a plurality of audio and video sub-segments with unique IDs; s2, for each audio and video sub-segment, three types of modal features including a voice semantic feature, a picture scene feature and an audio melody feature are extracted; according to the audio and video content intelligent retrieval method based on the multi-modal features, three types of core modal features are fused, voices, pictures and audio details of audios and videos are covered, the retrieval accuracy is improved to 92% or above, deviation of a single modal is avoided, three query modes of characters, voices and pictures are supported, and the retrieval efficiency is improved. The method can adapt to different scenes such as quick clip searching of short video platform users and accurate material positioning of media asset system administrators.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to an audio and video content intelligent retrieval method based on multi-modal features. BACKGROUND

[0002] Current audio and video content retrieval mainly relies on two ways: one is text label retrieval based on manual annotation, and the other is feature matching retrieval based on single modal (such as only picture or only audio), but the existing technology has obvious limitations: Manual annotation of text label has high cost and low efficiency, and is easily affected by subjective factors, leading to deviation between label and content, and unable to cover detailed information (such as objects in picture, melody changes in audio) in audio and video; Single modal retrieval has poor accuracy, for example, relying only on picture features cannot match the "birthday scene containing cheerful melody" type of demand, and relying only on audio features cannot locate the "red car appears in the picture" segment; Audio and video resources without labels or incomplete labels cannot be effectively retrieved, resulting in a large amount of high-quality content being deposited in the library and being difficult for users to find.

[0003] In view of the above problems, an intelligent retrieval method that can integrate multi-dimensional features and support multi-way query is urgently needed to break through the limitations of traditional technology and improve the efficiency and applicability of audio and video retrieval. SUMMARY

[0004] (I) Technical problems solved In view of the deficiencies of the prior art, the present application provides an audio and video content intelligent retrieval method based on multi-modal features, which solves the problems of traditional retrieval relying on manual labels, low accuracy of single modal, and narrow application scenarios, and realizes multi-feature fusion extraction, multi-way flexible query, and efficient and accurate matching of audio and video retrieval effect.

[0005] (II) Technical solutions To achieve the above purpose, the present application provides the following technical solutions: an audio and video content intelligent retrieval method based on multi-modal features, comprising the following steps: S1: Preprocessing the original audio and video data, including audio noise reduction, video frame extraction and screening, and audio and video segmentation, to obtain a plurality of audio and video sub-clips with unique ID; S2: For each audio and video sub-clip, extract three types of modal features: speech semantic features, picture scene features, and audio melody features; S3: Using attention mechanism weighted fusion strategy, normalize and weight distribute the three types of modal features to generate a unified multi-modal fusion feature vector; S4: Construct a database containing sub-fragment unique ID-multimodal fusion feature vector-original audio-video path, and construct a feature index library based on approximate nearest neighbor search technology; S5: Receive the user's text description, voice fragment or reference picture query request, convert the query request into a corresponding feature vector, calculate the similarity with the fusion feature vector in the index library, and return the target audio-video sub-fragment after sorting.

[0006] Preferably, the preprocessing of step S1 specifically includes: the audio preprocessing adopts spectral subtraction to remove noise, uniformly sets the audio format to PCM, and sets the sampling rate to 16 kHz; The video preprocessing extracts frames at 25 fps, removes blurred frames with a clarity score lower than 0.8, and uniformly sets the resolution to 1080P; The audio-video is divided into sub-fragments at 5-10 seconds per segment, and each sub-fragment is assigned a unique ID.

[0007] Preferably, the extraction process of the speech semantic feature in step S2 is: The speech in the audio is converted into text through automatic speech recognition technology, and the pre-trained BERT model is used to encode the text to obtain a speech semantic feature vector with a dimension of 768.

[0008] Preferably, the extraction process of the picture scene feature in step S2 is: One frame of key frame is taken every 2 seconds for the video sub-fragment, the YOLOv8 model is used to identify the human and object targets in the key frame, the ResNet50 model is used to classify the picture atmosphere, the target feature and the scene classification feature are spliced, and a picture scene feature vector with a dimension of 512 is obtained.

[0009] Preferably, the extraction process of the audio melody feature in step S2 is: The 39-dimensional MFCC feature of the audio is extracted using the Mel frequency cepstrum coefficient technology, the 39-dimensional MFCC feature contains first-order and second-order difference coefficients, the feature is reduced in dimension through a 3-layer convolutional neural network, and an audio melody feature vector with a dimension of 256 is obtained.

[0010] Preferably, the feature fusion of step S3 specifically includes: The three types of modal feature vectors are normalized to the [0, 1] interval; The contribution degrees of the three types of features are calculated based on historical retrieval data, dynamic attention weights are allocated, and the weights sum to 1; The three types of feature vectors are weighted and summed according to the attention weights to obtain a multimodal fusion feature vector with a dimension of 256.

[0011] Preferably, the index library construction of step S4 specifically includes: The unique ID of each sub-segment, the fused feature vector, and the original audio / video path are associated and stored in a MySQL database. An IVF_FLAT index (1000 cluster centers) is constructed using FAISS technology, and the index file is associated with the database; When the audio and video library data is updated, an incremental update of the index is automatically triggered.

[0012] Preferably, the text description retrieval process in step S5 is as follows: The BERT model is used to encode the text query input by the user to obtain the query semantic feature vector; Calculate the cosine similarity between the query vector and the fused feature vector in the index, and initially return the top 10 similarity sub-fragments; The initial results are sorted a second time based on users' historical preferences, and the top 5 search results and original audio / video links are displayed.

[0013] Preferably, the process of retrieving the speech segment in step S5 is as follows: For the voice query segment uploaded by the user, the voice semantic feature vector is obtained by the voice semantic feature extraction method in step S2; Calculate the cosine similarity between the query vector and the fused feature vector of the index, and return the retrieval results according to the sorting rules of claim 8.

[0014] Preferably, the reference image retrieval process in step S5 is as follows: For the reference image uploaded by the user, the feature vector of the query image is obtained by the image scene feature extraction method in step S2; Calculate the cosine similarity between the query vector and the fused feature vector of the index, and return the retrieval results according to the sorting rules of claim 8.

[0015] (III) Beneficial Effects Compared with existing technologies, this invention provides an intelligent audio and video content retrieval method based on multimodal features, which has the following beneficial effects: 1. This intelligent audio and video content retrieval method based on multimodal features, through multimodal feature fusion and FAISS indexing technology, shortens the retrieval response time to less than 0.5 seconds, improving efficiency by 3-5 times compared to traditional text tag retrieval. It integrates three core modal features, covering the voice, image, and audio details of audio and video, and improves the retrieval accuracy to over 92%, avoiding the bias of single modality. It supports three query methods: text, voice, and image, and can be adapted to different scenarios such as short video platform users quickly finding clips and media asset system administrators accurately locating materials.

[0016] 2. This intelligent audio and video content retrieval method based on multimodal features eliminates the need for manual labeling of a large number of text tags. It only requires model training on a small number of samples, reducing manual costs by more than 60%.

[0017] 3. Intelligent audio and video content retrieval methods based on multimodal features can further enhance retrieval capabilities by adding new modal features (such as subtitle features and motion features) or optimizing fusion algorithms, thus adapting to the future development of audio and video technologies. Detailed Implementation

[0018] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0019] The intelligent audio and video content retrieval method based on multimodal features includes the following steps: Step 1: Audio and video data preprocessing: The raw audio and video data undergoes standardization processing to ensure the accuracy of subsequent feature extraction. This includes: Audio preprocessing: Noise reduction algorithms, such as spectral subtraction, are used to remove environmental noise from the audio, and the audio format is unified to PCM format with a sampling rate of 16kHz. Video preprocessing: Frame extraction is performed on the video, the frame rate is set to 25fps, blurry frames are removed, and the video resolution is unified to 1080P by the frame clarity score. Data segmentation: The preprocessed audio and video are divided into several sub-segments of fixed duration, each lasting 5-10 seconds. Each sub-segment is assigned a unique ID for subsequent indexing and matching. Step 2, Multimodal Feature Extraction: For each audio / video sub-segment, three types of modal features are extracted: speech semantic features, visual scene features, and audio melody features. Speech semantic feature extraction: Automatic speech recognition (ASR) technology is used to convert speech in audio into text, and then a pre-trained semantic model (such as BERT) is used to encode the text to obtain a speech semantic feature vector with a dimension of 768. Scene feature extraction: For keyframes of video segments, one frame is taken every 2 seconds. An object detection model (such as YOLOv8) is used to identify targets such as people and objects in the scene. At the same time, a scene classification model (such as ResNet50) is used to classify the scene atmosphere, such as lively or quiet. The target features and scene classification features are concatenated to obtain a scene feature vector with a dimension of 512. Audio melody feature extraction: Mel frequency cepstral coefficients (MFCC) technology is used to analyze the audio, extracting 13-dimensional MFCC coefficients and first-order and second-order difference coefficients, for a total of 39-dimensional features. Then, the features are reduced in dimensionality using a convolutional neural network (CNN) to obtain an audio melody feature vector with a dimension of 256. Step 3: Multimodal feature fusion To eliminate dimensional differences and redundant information among features from different modalities, an attention-based weighted fusion strategy is employed to fuse the three types of feature vectors. Feature normalization: The feature vectors of speech semantics, scene image, and audio melody are normalized to the [0,1] interval to avoid the fusion effect being affected by the difference in numerical range; Attention weight allocation: Based on historical retrieval data, calculate the contribution of three types of features in different retrieval scenarios, such as finding dialogue clips or finding upbeat music clips, and assign dynamic attention weights to each type of feature (the weights sum to 1). Feature fusion generation: The three types of feature vectors are weighted and summed according to attention weights to obtain a unified multimodal fusion feature vector with a dimension of 256, which serves as the core retrieval feature of the audio and video sub-segment; Step 4: Construction of a unified feature index library: Based on multimodal fusion feature vectors, an efficient retrieval index library is constructed, specifically including: Feature storage: Store the unique ID, fused feature vector, and original audio / video path of each audio / video sub-segment in a database (such as MySQL); Index building: The Approximate Nearest Neighbor Search (ANNS) indexing technology (FAISS) is used to build an index structure for all fused feature vectors, and the index file is associated with the database to achieve the linkage between fast index location and accurate database data retrieval; Index Update: When new, deleted, or modified content is added to the audio and video library, an incremental update of the index is automatically triggered to ensure that the index is consistent with the actual data; Step 5: Multiple search and matching methods: Users can initiate searches using three methods: text description, voice clips, and reference images. The system completes the matching process as follows: Text description retrieval: Users input text queries, such as a clip with a cat in the video and cheerful background music. The BERT model is used to encode the text to obtain the query semantic feature vector. The cosine similarity is calculated with the fused feature vector in the index, and the top 10 audio and video clips with the highest similarity are returned. Voice segment retrieval: Users upload voice query segments, such as "Hello, how's the weather today?" The voice semantic feature extraction method in step 2 is used to obtain the query voice feature vector. The feature vector is then fused with the index to calculate the similarity and return the matching result. Reference image retrieval: Users upload reference images, such as a red sunset scene. The feature vector of the query image is obtained by the image scene feature extraction method in step 2. The feature vector is then fused with the index to calculate the similarity and return the matching result. Results ranking: For the matched sub-segments, combined with the user's historical preferences, such as the pet-related content that the user frequently clicks on, the pet-related segments are ranked higher. The similarity results are then ranked a second time, and finally the top 5 search results are displayed to the user along with the corresponding original audio and video links.

[0020] Example: Implementation environment: Hardware environment: The server uses an Intel Xeon Gold 6348 CPU, an NVIDIA A100 GPU, 128GB of memory, and 10TB of SSD storage; Software environment: The operating system is Linux CentOS 8.0, the deep learning framework is PyTorch 2.0, the database is MySQL 8.0, and the index library is FAISS 1.7.4.

[0021] The implementation steps are as follows: Step 1: Data preprocessing: Obtain 1 million original short videos (15-60 seconds in length) from short video platforms, remove audio noise using spectral subtraction, and unify the audio format to PCM (16kHz). The video was framed at 25fps, and blurry frames with a resolution lower than 0.8 were removed, with a uniform resolution of 1080P. Each video is divided into 5-second segments, resulting in 4 million segments, each assigned a unique ID (e.g., VID_20240501_00001).

[0022] Step 2, Multimodal Feature Extraction: Speech and semantic features: Speech is converted to text using Alibaba Cloud ASR, and the text is encoded using pre-trained BERT (Chinese version) to obtain a 768-dimensional feature vector; Scene features: Take one key frame every 2 seconds, use YOLOv8n (lightweight version) to identify the target, use ResNet50 to classify the scene, and concatenate to obtain a 512-dimensional feature vector; Audio melody features: 39-dimensional MFCC features were extracted and reduced to 256-dimensional feature vectors through a 3-layer CNN.

[0023] Step 3, Feature Fusion: After normalizing the three types of features, based on the platform's historical data (search records of the past 3 months), initial weights (0.3, 0.4, 0.3) are assigned to speech semantics, visual scene, and audio melody. The weights are dynamically adjusted through an attention mechanism. For example, when a user searches for a singing clip, the speech semantic weight is increased to 0.5, resulting in a 256-dimensional fused feature vector.

[0024] Step 4: Index building: The sub-segment ID, fusion feature, and video path are stored in MySQL. An IVF_FLAT index (with 1000 cluster centers) is built using FAISS. The index file size is approximately 50GB. After being associated with the database, it supports more than 1000 retrieval requests per second.

[0025] Step 5, Search Test: When a user enters text to search for a segment featuring a puppy and upbeat music, the system encodes the text to obtain a query vector, matches it against the index, and returns 10 matching segments within 0.3 seconds. After a second sorting, the top 5 segments are displayed. A user uploads a 10-second voice message about going to the park today. The system converts the message to text, matches it, and returns a short video clip containing the voice message. Users upload reference images of a blue sky, the system extracts the features of the image, and returns a short video clip containing the blue sky. Test results: The search accuracy rate was 93.5%, and the probability of users clicking on the matching results was 88%, which is a significant improvement compared to the platform's original text tag search (accuracy rate of 72%).

[0026] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An intelligent retrieval method for audio-video content based on multi-modal features, comprising the following steps: S1: Preprocessing the original audio-video data, including audio noise reduction, video frame extraction and screening, and audio-video segmentation, to obtain a plurality of audio-video sub-clips with unique IDs; S2: For each audio-video sub-clip, extracting three types of modal features, namely speech semantic features, picture scene features, and audio melody features; S3: Using an attention mechanism weighting fusion strategy to normalize and assign weights to the three types of modal features, generating a unified multi-modal fusion feature vector; S4: Constructing a database containing sub-clip unique ID-multi-modal fusion feature vector-original audio-video path, and building a feature index library based on approximate nearest neighbor search technology; S5: Receiving user's text description, voice clip or reference picture query request, converting the query request into corresponding feature vector, calculating the similarity with the fusion feature vector in the index library, and returning the target audio-video sub-clip after sorting. 2.The method of claim 1, wherein the method further comprises: The preprocessing of step S1 specifically includes: Audio preprocessing uses spectral subtraction to remove noise, and the audio format is unified to PCM with a sampling rate of 16kHz; Video preprocessing extracts frames at 25fps, removes blurred frames with a clarity score below 0.8, and unifies the resolution to 1080P; Audio-video is divided into sub-clips at 5-10 seconds per segment, and each sub-clip is assigned a unique ID. 3.The method of claim 1, wherein: The extraction process of speech semantic features in step S2 is: Convert the speech in the audio to text through automatic speech recognition technology, and encode the text using a pre-trained BERT model to obtain a speech semantic feature vector with a dimension of 768. 4.The method of claim 1, wherein the method further comprises: The extraction process of picture scene features in step S2 is: Take one key frame every 2 seconds for the video sub-clip, use the YOLOv8 model to identify the human and object targets in the key frame, use the ResNet50 model to classify the picture atmosphere, and concatenate the target features and scene classification features to obtain a picture scene feature vector with a dimension of 512. 5.The method of claim 1, wherein: The extraction process of audio melody features in step S2 is: Use the Mel-frequency cepstrum coefficient (MFCC) technique to extract 39-dimensional MFCC features from the audio, which contain first-order and second-order difference coefficients. Reduce the dimension of the features through a 3-layer convolutional neural network to obtain an audio melody feature vector with a dimension of 256. 6.The method of claim 1, wherein the method further comprises: determining a plurality of feature vectors of the audio-video content based on a plurality of feature extraction algorithms; and determining a plurality of feature vectors of the query based on the plurality of feature extraction algorithms. The feature fusion of step S3 specifically includes: Normalize the three types of modal feature vectors to the [0, 1] interval; Calculate the contribution of the three types of features based on historical retrieval data, assign dynamic attention weights, and the sum of the weights is 1; Weighted sum of the three types of feature vectors according to the attention weights to obtain a multi-modal fusion feature vector with a dimension of 256. 7.The method of claim 1, wherein: The index library construction of step S4 specifically includes: Store the sub-clip unique ID-fusion feature vector-original audio-video path in the MySQL database; Use the FAISS technology to build an IVF_FLAT index (cluster center number 1000), and associate the index file with the database; When the audio-video library data is updated, automatically trigger the incremental update of the index library. 8.The method of claim 1, wherein the method further comprises: determining a plurality of feature vectors of the audio-video content based on a plurality of feature extraction algorithms; and determining a plurality of feature vectors of the query based on the plurality of feature extraction algorithms. The process of text description retrieval in step S5 is: The BERT model is used to encode the character query input by the user to obtain a query semantic feature vector; A cosine similarity of the query vector and the fusion feature vector in the index library is calculated, and a sub-fragment with a similarity of Top10 is returned initially; The initial result is sorted again in combination with the user's historical preferences, and the retrieval result and original audio / video jump link of Top5 are displayed. 9.The method of claim 1, wherein: The process of the voice segment retrieval in step S5 is as follows: The query voice feature vector is obtained from the voice query segment uploaded by the user according to the voice semantic feature extraction method in step S2; The cosine similarity of the query vector and the fusion feature vector in the index library is calculated, and the retrieval result is returned according to the sorting rule in claim 8. 10.The method of claim 1, wherein the method further comprises: determining a plurality of feature vectors of the audio-video content based on a plurality of feature extraction algorithms; and determining a plurality of feature vectors of the query based on the plurality of feature extraction algorithms. The process of the reference picture retrieval in step S5 is as follows: The query picture feature vector is obtained from the reference picture uploaded by the user according to the picture scene feature extraction method in step S2; The cosine similarity of the query vector and the fusion feature vector in the index library is calculated, and the retrieval result is returned according to the sorting rule in claim 8.