Video search method and system based on video content recognition algorithm

Abstract

The invention discloses a video search method and system based on a video content recognition algorithm, and the method comprises the steps: carrying out the format conversion and framing processing of an input video file, and storing an extracted video frame; extracting multi-modal features in the video frame by adopting a video content understanding algorithm; storing the extracted video feature information into a reverse index data structure, establishing a video meta-database, and recording metadata information of the video; the query statement input by the user is understood and analyzed, and related video features and video metadata information are retrieved according to query content; calculating the correlation between the retrieved video and the user query; and sorting all retrieved videos according to the correlation to obtain a final video search result. The method has the advantages that the limitation that a traditional method only depends on a single feature for searching is overcome, user intentions are accurately grasped, user query can be processed in a short time, results can be returned, the accuracy of video searching is improved, and user requirements are better matched.

Description

Technical Field

[0001] The present invention relates to the field of information retrieval and computer vision technology, and in particular to a video search method and system based on a video content recognition algorithm. Background Art

[0002] Traditional methods only focus on a single feature of the video when searching for videos, such as searching based on visual features such as color and texture, while ignoring multimodal information such as audio features, object detection, and scene recognition, resulting in low accuracy when searching for videos with complex content or specific scenes. The index results used by some methods have low retrieval efficiency under large-scale video data and cannot quickly respond to user queries. Summary of the invention

[0003] The purpose of the present invention is to provide a video search method and system based on a video content recognition algorithm, so as to solve the above-mentioned problems existing in the prior art.

[0004] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0005] A video search method based on a video content recognition algorithm comprises the following steps:

[0006] S1. Video preprocessing:

[0007] Perform format conversion and frame division processing on the input video file, and store the extracted video frames;

[0008] S2. Video feature extraction:

[0009] A video content understanding algorithm that uses multiple algorithms to work together to extract multimodal features from video frames;

[0010] S3, video index and video metadata database establishment:

[0011] The extracted video feature information is stored in an inverted index data structure, and a video metadata database is established to record the metadata information of the video;

[0012] S4. User query processing:

[0013] Understand and analyze the query statements entered by the user, and retrieve relevant video features and video metadata information based on the query content;

[0014] S5. Video matching and sorting:

[0015] Calculate the correlation between the retrieved videos and the user query; and sort all retrieved videos according to the correlation to obtain the final video search results.

[0016] Preferably, step S1 specifically includes the following contents:

[0017] S11, input the video, use the video format conversion question to convert the video into a widely supported format, and adjust the corresponding parameters during the conversion process to ensure that the quality and size of the converted video meet the requirements;

[0018] S12, reading the video, and determining the number of frames per second according to the frame rate of the video;

[0019] S13, extracting video frames from the video at a certain time interval or frame number interval, and storing the extracted video frames.

[0020] Preferably, in step S13, the video frames are saved in a specific folder in the form of image files, or the video frames are stored in a memory in the form of data structures.

[0021] Preferably, step S2 specifically includes the following contents:

[0022] S21, selecting a suitable object detection model, using the object detection model to identify and locate objects in the video frame, and outputting the detected object category and bounding box;

[0023] S22, using a scene recognition model to determine the scene category to which the video frame belongs, and outputting a scene category label;

[0024] S23, performing face detection on the video frame to determine whether there is a face in the video frame, and outputting the position coordinates of the face;

[0025] S24. Extract the audio track from the video, and perform speech recognition and audio feature extraction on the audio.

[0026] Preferably, video index establishment specifically includes the following contents:

[0027] S311, encoding or serializing the extracted various video features;

[0028] S312, constructing an inverted index data structure; taking each video feature as an index item and the corresponding video frame as an index value;

[0029] S313: Store the encoded feature information in an inverted index data structure.

[0030] Preferably, the establishment of the video metadata database specifically includes the following contents:

[0031] S321, determining the video metadata information to be recorded, including basic information such as the video title, description, and time, and technical information such as format, resolution, frame rate, and encoding method;

[0032] S322: For each video, extract metadata information of the video;

[0033] S323: Create a record for each video in the database, where the record contains a unique identifier of the video, video metadata information, and an association with an inverted index data structure.

[0034] Preferably, step S4 specifically includes the following contents:

[0035] S41, when a user inputs a query statement, the query statement is analyzed using a natural language processing method to understand the intent of the query;

[0036] S42. For complex queries, use named entity recognition methods to extract specific entity information in the query;

[0037] S43. According to the understood query content, search in the video index, use the inverted index data structure to quickly find the video feature information related to the query; at the same time, search in the video metadata database to obtain the metadata information of the video.

[0038] Preferably, step S5 specifically includes the following contents:

[0039] S51. For the retrieved video, use a deep learning-based method to calculate the correlation between the video features and the user query;

[0040] S52. Use a sorting algorithm to efficiently sort all retrieved videos according to the relevance scores, put videos with high scores at the front, and obtain the final video search results.

[0041] Preferably, the sorted video search results are displayed to the user, and relevant preview and playback functions are provided; based on user feedback, the performance of the video search method is continuously optimized.

[0042] The present invention also aims to provide a video search system based on a video content recognition algorithm, the system can implement the above-mentioned method, the system comprises:

[0043] Video preprocessing module: converts the format and frames of the input video files, and stores the extracted video frames;

[0044] Video feature extraction module: uses a video content understanding algorithm that uses multiple algorithms to extract multimodal features from video frames;

[0045] Video indexing and video metadata database establishment module: stores the extracted video feature information into an inverted index data structure, and establishes a video metadata database to record the metadata information of the video;

[0046] User query processing module: understands and analyzes the query statements input by users, and retrieves relevant video features and video metadata information based on the query content;

[0047] Video matching and sorting module: calculates the correlation between the retrieved videos and the user query; and sorts all retrieved videos according to the correlation to obtain the final video search results.

[0048] The beneficial effects of the present invention are: the present invention overcomes the limitation of traditional methods that only rely on a single feature for search, accurately grasps user intentions, establishes an efficient index structure and retrieval algorithm, can process user queries and return results in a short time, improves the accuracy of video search, and thus better matches user needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of a search method in an embodiment of the present invention;

[0050] Figure 2 is a flow chart of establishing a video index and a metadata database in an embodiment of the present invention;

[0051] Figure 3 4 is a flowchart of user query processing in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation methods described herein are only used to explain the present invention and are not used to limit the present invention.

[0053] like Figure 1 As shown, in this embodiment, a video search method based on a video content recognition algorithm is provided, including the following parts:

[0054] 1. Video Preprocessing

[0055] Convert the input video file into different formats and perform frame processing, and store the extracted video frames. This includes the following:

[0056] 1.1. Format conversion: Input the video and use the video format conversion tool to convert it into a widely supported format, such as MP4, AVI, MOV, etc. During the conversion, some parameters need to be adjusted, such as video resolution, frame rate, encoding method, etc., to ensure that the quality and size of the converted video meet the requirements.

[0057] 1.2. Determine the number of frames: Read the video file, obtain the frame rate, duration and other information of the video, and determine the number of frames per second based on the frame rate. For example, 24 frames per second means that each second of video can be divided into 24 separate video frames.

[0058] 1.3. Frame processing: Extract video frames from the video at a certain time interval or frame interval, and store the extracted video frames. They can be saved in a specific folder as image files, or stored in the memory in the form of a data structure for subsequent processing.

[0059] 2. Video Feature Extraction

[0060] Advanced video content understanding algorithms such as deep learning are used to extract multimodal features such as visual features and audio features from video frames. These features can include information such as object detection, scene recognition, face detection, and speech analysis. Video content understanding algorithms usually require the synergy of multiple algorithms. The specific extraction process is as follows:

[0061] 2.1 Object detection: Select a suitable object detection algorithm, such as a deep learning-based target detection model such as YOLO, Faster R-CNN, etc., input the video frame into the object detection model, the model recognizes and locates the objects in the image, and outputs the detected object category, bounding box and other information.

[0062] 2.2. Scene recognition: Input the video frame into the scene recognition model. The model will determine the scene category to which the image belongs, such as indoor, outdoor, urban, rural, etc., and output the scene category label.

[0063] 2.3. Face detection: Perform face detection on the video frame to determine whether there is a face in the image and output the location coordinates of the face.

[0064] S24, Speech Analysis: Extract the audio track from the video and analyze the audio, including speech recognition, audio feature extraction, etc. Speech recognition can convert the speech content in the audio into text for further analysis; audio feature extraction can extract the spectral features of the audio, Mel Frequency Cepstral Coefficients (MFCC), etc.

[0065] 3. Establishment of Video Index and Video Metadata Database

[0066] like Figure 2 As shown, it includes two parts: first, the extracted video feature information is stored in a data structure such as an inverted index for fast retrieval. Second, a video metadata database is established to record video metadata information such as the title, description, and time of the video.

[0067] 3.1. Video index establishment specifically includes the following contents:

[0068] (1) Feature encoding: The various video features extracted in the second step, such as object detection results, scene recognition labels, face detection information, etc., are encoded or serialized for storage and retrieval.

[0069] (2) Construct an inverted index: Construct an inverted index data structure, where each feature is used as an index item and the corresponding video frame or video clip is used as the index value. For example, if there are multiple frames in a video where the object "car" is detected, then the feature "car" will correspond to the index of these frames.

[0070] (3) Feature storage: The encoded feature information is stored in the inverted index so that video frames or video clips containing specific features can be quickly found during retrieval.

[0071] 3.2. The establishment of the video metadata database specifically includes the following contents:

[0072] (1) Determine metadata: Determine the video metadata information that needs to be recorded, including basic information such as the video title, description, time, and technical information such as format, resolution, frame rate, and encoding method.

[0073] (2) Extract metadata information: For each video, extract the metadata information. You can obtain the time information from the attributes of the video file, manually enter the title and description, or obtain it from relevant data sources.

[0074] (3) Storing metadata: Use a relational database or a non-relational database for management and query. In the database, a record can be created for each video, containing the video's unique identifier, metadata information, and an association with an inverted index so that detailed information and features of a specific video can be quickly retrieved.

[0075] 4. User query processing:

[0076] Receive natural language queries from users, understand and analyze the query statements entered by users, and retrieve relevant video features and video metadata information based on the query content. Figure 3 As shown, it specifically includes the following contents:

[0077] 4.1. Query understanding: When a user enters a query statement, the query statement needs to be analyzed using natural language processing (NLP) to perform lexical analysis, syntactic analysis, semantic analysis, etc. to understand the intent of the query. For example, the query statement can be broken down into keywords, phrases, or sentence structures to better understand the user's needs.

[0078] 4.2. Named Entity Recognition: For complex queries, named entity recognition technology can be used to extract specific entity information, such as names of people, places, and organization names.

[0079] 4.3. Retrieve videos: Based on the understood query content, search the video index established in step 3, and use the inverted index data structure to quickly find video feature information related to the query. For example, if the user queries "videos containing cars", you can find video frames containing "cars" in the object detection features.

[0080] 4.4. Retrieve metadata: At the same time, search the video metadata database established in the third step to obtain metadata information such as the video title, description, time, etc. These metadata can help further filter and understand the videos related to the query.

[0081] 5. Video matching and sorting:

[0082] Calculate the correlation between the retrieved videos and the user query; and sort all the retrieved videos according to the correlation to obtain the final video search results. Specifically, it includes the following contents:

[0083] 5.1. Correlation calculation: For the retrieved videos, a deep learning-based method is used to calculate the correlation score between the video features and the user query. The calculation process is to encode the video and query using a pre-trained model, and then calculate the direct similarity of the encoded vectors.

[0084] 5.2. Video Sorting: Use a sorting algorithm to efficiently sort the retrieved videos based on their similarity scores, with videos with higher scores being ranked first.

[0085] 6. Search Results Presentation

[0086] The sorted video search results are displayed to users, and relevant preview, playback and other functions are provided. Based on user feedback, the video search algorithm and system performance are continuously optimized.

[0087] In this embodiment, a video search system based on a video content recognition algorithm is provided. The system can implement the above-mentioned method. The system includes:

[0088] (1) Video preprocessing module: performs format conversion and frame segmentation on the input video file, and stores the extracted video frames;

[0089] (2) Video feature extraction module: This module uses a video content understanding algorithm that uses multiple algorithms to extract multimodal features from video frames.

[0090] (3) Video indexing and video metadata database establishment module: storing the extracted video feature information into an inverted index data structure, and establishing a video metadata database to record the metadata information of the video;

[0091] (4) User query processing module: understands and analyzes the query statement input by the user, and retrieves relevant video features and video metadata information based on the query content;

[0092] (5) Video matching and sorting module: Calculates the correlation between the retrieved videos and the user query; and sorts all retrieved videos according to the correlation to obtain the final video search results.

[0093] By adopting the above technical solution disclosed in the present invention, the following beneficial effects are obtained:

[0094] The present invention provides a video search method and system based on a video content recognition algorithm. The present invention overcomes the limitation of traditional methods that only rely on a single feature for search, accurately grasps user intentions, establishes an efficient index structure and retrieval algorithm, can process user queries and return results in a short time, improves the accuracy of video search, and thus better matches user needs.

[0095] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be considered as the scope of protection of the present invention.

Claims

1. A video search method based on a video content recognition algorithm, characterized in that: The following steps are included: S1. Video preprocessing: Perform format conversion and frame division processing on the input video file, and store the extracted video frames; S2. Video feature extraction: A video content understanding algorithm that uses multiple algorithms to work together to extract multimodal features from video frames; S3, video index and video metadata database establishment: The extracted video feature information is stored in an inverted index data structure, and a video metadata database is established to record the metadata information of the video; S4. User query processing: Understand and analyze the query statements entered by the user, and retrieve relevant video features and video metadata information based on the query content; S5. Video matching and sorting: Calculate the correlation between the retrieved videos and the user query; and sort all retrieved videos according to the correlation to obtain the final video search results.

2. The video search method based on the video content recognition algorithm according to claim 1, characterized in that: Step S1 specifically includes the following contents: S11, input the video, use the video format conversion question to convert the video into a widely supported format, and adjust the corresponding parameters during the conversion process to ensure that the quality and size of the converted video meet the requirements; S12, reading the video, and determining the number of frames per second according to the frame rate of the video; S13, extracting video frames from the video at a certain time interval or frame number interval, and storing the extracted video frames.

3. The video search method based on video content recognition algorithm according to claim 2 is characterized in that: In step S13, the video frame is saved in a specific folder in the form of an image file, or the video frame is stored in a memory in the form of a data structure.

4. The video search method based on video content recognition algorithm according to claim 1, characterized in that: Step S2 specifically includes the following contents: S21, selecting a suitable object detection model, using the object detection model to identify and locate objects in the video frame, and outputting the detected object category and bounding box; S22, using a scene recognition model to determine the scene category to which the video frame belongs, and outputting a scene category label; S23, performing face detection on the video frame to determine whether there is a face in the video frame, and outputting the position coordinates of the face; S24. Extract the audio track from the video, and perform speech recognition and audio feature extraction on the audio.

5. The video search method based on video content recognition algorithm according to claim 1, characterized in that: The video index establishment specifically includes the following contents: S311, encoding or serializing the extracted various video features; S312, constructing an inverted index data structure; taking each video feature as an index item and the corresponding video frame as an index value; S313: Store the encoded feature information in an inverted index data structure.

6. The video search method based on video content recognition algorithm according to claim 1, characterized in that: The establishment of the video metadata database specifically includes the following contents: S321, determining the video metadata information to be recorded, including basic information such as the video title, description, and time, and technical information such as format, resolution, frame rate, and encoding method; S322: For each video, extract metadata information of the video; S323: Create a record for each video in the database, where the record contains a unique identifier of the video, video metadata information, and an association with an inverted index data structure.

7. The video search method based on video content recognition algorithm according to claim 1, characterized in that: Step S4 specifically includes the following contents: S41, when a user inputs a query statement, the query statement is analyzed using a natural language processing method to understand the intent of the query; S42. For complex queries, use named entity recognition methods to extract specific entity information in the query; S43. According to the understood query content, search in the video index, use the inverted index data structure to quickly find the video feature information related to the query; at the same time, search in the video metadata database to obtain the metadata information of the video.

8. The video search method based on video content recognition algorithm according to claim 1, characterized in that: Step S5 specifically includes the following contents: S51. For the retrieved video, use a deep learning-based method to calculate the correlation between the video features and the user query; S52. Use a sorting algorithm to efficiently sort all retrieved videos according to the relevance scores, put videos with high scores at the front, and obtain the final video search results.

9. The video search method based on video content recognition algorithm according to claim 1, characterized in that: Display the sorted video search results to users and provide relevant preview and playback functions; Based on user feedback, the performance of the video search method is continuously optimized.

10. A video search system based on a video content recognition algorithm, characterized in that: The system can implement the method described in any one of claims 1 to 9, and the system includes: Video preprocessing module: converts the format and frames of the input video files, and stores the extracted video frames; Video feature extraction module: uses a video content understanding algorithm that uses multiple algorithms to extract multimodal features from video frames; Video indexing and video metadata database establishment module: stores the extracted video feature information into an inverted index data structure, and establishes a video metadata database to record the metadata information of the video; User query processing module: understands and analyzes the query statements input by users, and retrieves relevant video features and video metadata information based on the query content; Video matching and ranking module: calculates the relevance between the retrieved videos and the user query; And sort all the retrieved videos according to their relevance to get the final video search results.

Images