An automatic labeling and acquisition standardization method and system based on video content

Abstract

An automatic labeling and acquisition standardization method and system based on video content, including: decoding the video file into a processable video frame sequence, performing standardization processing, unifying format and encoding standard, adjusting resolution and frame rate; performing content analysis on the standardized video frame sequence, extracting key frames and identifying objects, scenes and actions, generating video analysis results and preliminary labels; according to the video content analysis results, separating the audio stream from the video for noise reduction processing and speech-to-text analysis, combining the processed text with the video content analysis results to generate comprehensive multi-modal labeling information; based on the comprehensive labeling information, taking the user's feedback in video retrieval, label clicking and watching behavior as the basis for adjusting the weight of the label, updating the weight distribution of the label by analyzing the user behavior data; integrating the labeling information and weight distribution into the index database, and updating the index in real time according to the user behavior and content changes.

Description

Technical Field

[0001] The present invention belongs to the field of computer Internet technology, and more specifically, relates to a method and system for automatic annotation and acquisition standardization based on video content. Background Art

[0002] With the rapid development of the internet and social media, the generation and accumulation of video content has increased dramatically, resulting in the emergence of massive amounts of video data. Due to the complexity and diversity of video content, manual annotation and management of these videos is both time-consuming and costly. Therefore, automated video annotation and collection methods are urgently needed to improve processing efficiency. In particular, without effective annotation, users searching for specific content on video platforms often face difficulties in information retrieval and low search efficiency.

[0003] Existing video annotation methods primarily rely on traditional manual annotation or simple content analysis to generate tags. These methods often fail to accurately capture the temporal nature of videos, scene changes, or the semantic connections between video and audio content, resulting in low annotation quality and efficiency. For video platforms and social media, the lack of efficient annotation and indexing significantly reduces the effectiveness of recommendation algorithms, making it difficult to accurately analyze and meet user preferences.

[0004] Furthermore, with the increasing demand for video content review, platforms and content providers require a more efficient tagging system to promptly update content and ensure that platform content complies with relevant regulations. Existing tagging systems often lack dynamic adjustment capabilities and are unable to provide real-time optimization when content is updated or user behavior changes, impacting platform content management and user experience.

[0005] Therefore, the market urgently needs a method that can effectively combine the visual features, audio semantics and user behavior of videos to achieve efficient labeling, dynamic optimization and intelligent recommendation of video content. Summary of the Invention

[0006] In order to solve the deficiencies in the prior art, the present invention provides a method and system for automatic annotation and acquisition standardization based on video content.

[0007] The present invention adopts the following technical solutions.

[0008] A first aspect of the present invention provides a method for automatic annotation and acquisition standardization based on video content, comprising the following steps:

[0009] Step 1: decode the original video file into a video frame sequence, adjust the video frames according to the preset resolution and frame rate, and perform standardization processing;

[0010] Step 2: The standardized video frame sequence is divided into multiple time periods according to the time sequence of the frames. In each time period, the content change between adjacent video frames is calculated, and the frame with the largest content change is selected as the key frame. At the same time, video semantic information is generated for each key frame, including a spatiotemporal identifier and a preliminary label. The spatiotemporal identifier is used to record the time position and segment range of the key frame; the preliminary label is used to record the scene type, object, and action of the key frame.

[0011] Step 3: Extract the audio portion from the original video file to generate audio data synchronized with the video frame time; preprocess the extracted audio data, convert the preprocessed audio data into text using speech recognition, extract the semantic information in the audio, and generate corresponding text features; associate and fuse the text features with the spatiotemporal identifiers and preliminary labels of the key frames to generate a multimodal annotation containing video semantics, audio semantics, and timeline information;

[0012] Step 4: Based on multimodal annotation, during the user's interaction with the video content, real-time user behavior data is collected, the tags clicked by the user are associated with the corresponding spatiotemporal identifiers, and the user's interests and preferences are analyzed. Based on the analysis results, the keyframe extraction criteria are dynamically updated, including adjusting the threshold of the content change degree, optimizing the tag classification rules, and adjusting the weight ratio of multiple modalities to optimize the automatic annotation and collection standardization process of video content;

[0013] Step 5: Integrate the dynamically updated key frames, tags, and multimodal annotation information into the index database, and make local adjustments to the content and structure of the index database in real time according to changes in user behavior and video content. When the data changes, update the affected data part, adjust the index structure according to the user's retrieval and access frequency, establish access channels for tags and key frames with access frequencies higher than the set value, adjust the sorting and priority of index fields, add index items related to user interests, delete redundant index information and index information with access frequencies lower than the set value, so that the index database is updated synchronously with the video content and user behavior data.

[0014] Preferably, step 1 comprises:

[0015] Decode the original video file into a sequence of separate video frames, so that each frame can be processed as an independent data unit;

[0016] Convert the decoded video frame sequence into a consistent image format. Through distributed computing, assign different video frames to different computing nodes. Select the optimal encoding standard based on the computing power and network bandwidth of each node, so that all video frames have the same file format and encoding standard.

[0017] Standardize the resolution of video frames and scale them using an interpolation algorithm to adjust the resolution of all frames to the same standard. When adjusting the resolution, the aspect ratio of the original video is maintained, and black borders and cropping are used to match the target video frame resolution.

[0018] The frame rate of the video frames is standardized. When the frame rate of the video frames does not meet the target frame rate and the number of frames does not match the target requirements, some frames are removed to meet the target frame rate requirements; if the number of frames does not meet the target requirements, intermediate frames are generated through interpolation.

[0019] Preferably, step 2 includes:

[0020] Content change detection is performed on standardized video frame sequences. The video frame sequence is divided into multiple time periods based on the timestamp information of the video frames. The pixel differences and color histogram changes between the video frames are analyzed. Based on a preset threshold, frames with content changes exceeding the threshold are screened out in each time period. These frames are regarded as the most representative frames and marked as key frames.

[0021] Generate a spatiotemporal identifier for each selected keyframe, which records the time position of the keyframe and the time range of the segment it is in, and establishes an association between the keyframe and the global timeline;

[0022] Identify and classify objects in the image in key frames, use bounding boxes to mark object locations and category information, and perform deduplication and screening on identified objects;

[0023] Identify and label different objects, scenes, and actions in keyframes and integrate them into preliminary labels, including:

[0024] Perform object detection and classification on each frame, mark the object location and category information with a rectangular box, remove duplicates and filter out the objects that best represent the overall features, and generate corresponding object labels;

[0025] Extract color, texture, and shape features from keyframes, identify scene types using a pre-trained scene classification model, label and classify the identified scenes, and generate corresponding scene labels. Integrate scene information from consecutive keyframes to ensure consistency across all annotations.

[0026] Extract human key points, posture information, and motion trajectories from keyframes, identify motion types using a pre-trained motion classification model, label each motion, and attach the motion labels to the corresponding keyframes for semantic analysis and collection.

[0027] The spatiotemporal identifier records the time position of the keyframe and the time range of the corresponding fragment to establish the association between the keyframe and the global timeline; the preliminary label is generated based on the visual content of the keyframe, including scene type, object features and action description, providing preliminary semantic information for the keyframe.

[0028] Preferably, the performing content change detection on the standardized video frame sequence includes:

[0029] By calculating the pixel differences and color histogram changes between adjacent frames, the degree of change in the frame content is determined;

[0030] The video is segmented into different scenes by analyzing the differences between frames, including: setting a frame threshold, calculating the difference between consecutive frames, and considering the current frame as a key frame if the difference exceeds the preset threshold, and comparing the color histograms of consecutive frames. If the difference between the histograms exceeds the threshold, the current frame is considered a key frame.

[0031] Preferably, step 3 includes:

[0032] Based on the video content analysis results, separate the audio stream from the video file, extract the audio stream into an independent file, and convert it to a consistent sampling rate and bit depth;

[0033] Apply frequency-domain filtering and time-domain filtering to the audio stream for noise reduction. This includes extracting frequency-domain features through fast Fourier transform and removing background noise using frequency-domain filters. Furthermore, adaptive filtering and delay cancellation algorithms are combined to provide clear speech input for the speech-to-text process.

[0034] Perform speech-to-text processing on the noise-reduced audio, using a pre-trained automatic speech recognition model to convert speech content into text, recognize the speech content, and output text information. Set a recognition confidence threshold and filter out transcripts below the threshold.

[0035] The generated text is processed, including stop word removal, stemming, and synonym merging. The text content is analyzed for sentiment, the emotional information expressed in the speech is identified, and sentiment tags are generated to provide semantic information for the video content.

[0036] The processed text content and sentiment analysis results are combined with the object, scene and action labels of the video. The text information generated by the audio content is associated with the corresponding video frames and key frame time points according to the key frame time positions recorded in the spatiotemporal identifier. Through the integration of multimodal information, a comprehensive label containing objects, scenes, actions, text and emotions is formed to generate multimodal comprehensive annotation information.

[0037] Preferably, performing speech-to-text processing on the audio after noise reduction processing includes:

[0038] The voice signal is decomposed into multiple frequency subbands, and voice features are extracted on different frequency subbands respectively, and the features of different frequency subbands are fused by introducing a multi-level neural network structure.

[0039] Preferably, step 4 comprises:

[0040] Based on multi-modal annotation, when the user interacts with the video content, the user's search, label clicking and watching behavior data are collected, the user's search keywords, the number of clicked labels, the watching time and the playback frequency are recorded, and stored in the user behavior database;

[0041] According to the label clicked by the user, each label is associated with the corresponding spatiotemporal identifier, and the spatiotemporal identifier records the time position and segment range of the key frame in the video. By analyzing the label clicked by the user and the corresponding spatiotemporal identifier, the user's preference and interest are obtained and the label weight is calculated;

[0042] According to the analysis result, the key frame extraction standard is dynamically updated, including adjusting the threshold of content change degree, optimizing the label classification rule, and distinguishing the priority;

[0043] According to the priority, the weight proportion of multi-modal fusion is adjusted, and according to the attention degree of the user to the video semantic and audio semantic, the parameter setting of the fusion algorithm is dynamically adjusted;

[0044] The updated label weight is synchronized to the index database, and is applied in the search and recommendation function in real time, and the weight change history of the label is reserved, so that the latest weight data is used when the user queries next time, and the automatic annotation and standardized process of video content are optimized.

[0045] Preferably, step 5 comprises:

[0046] According to the weight adjustment result, the weight data of the label and the corresponding video content annotation information are integrated into the index database, the index database includes the comprehensive annotation information of the objects, scenes, actions, texts and emotions of the video, and each label is assigned a weight;

[0047] Set the index structure, including the main index table, the label association table, the label association table, the audio index table and the video frame index table; all the associated labels and spatiotemporal information in the label system are stored in the database in a structured form, and each video entry in the database contains a series of index items, including object labels, scene labels, action labels, text labels and spatiotemporal information associated with the labels;

[0048] An update mechanism of the index database is established, when the data changes, the affected data part is updated, and the whole database is not updated;

[0049] Adjust the index structure based on the user's search and access frequency, establish access channels for tags and keyframes with access frequencies higher than the set value, and adjust the sorting and priority of index fields;

[0050] Add index items related to user interests and delete redundant index information or index information with access frequency lower than the set value; the set value range is the number of access times of tags or keyframes that ranks in the top 10% to 30% of the total access volume; the time range is the time window used to count user behavior, which is the last 7 to 30 days starting from the current time;

[0051] Synchronously update the index database with video content and user behavior data to achieve real-time local adjustments to meet users' personalized needs.

[0052] Preferably, establishing an update mechanism for the index database includes:

[0053] When a local change in a video file is detected, the index related to the changed part is updated without re-indexing the entire video file;

[0054] Divide video content and index data into multiple independent content blocks, each of which can be independently indexed and updated; update the index in real time through user interactions, including tagging, video comments, and content suggestions;

[0055] Dynamically adjust the index update direction based on users' real-time search and viewing behaviors, giving priority to updating relevant video indexes that users are interested in;

[0056] The index is updated collaboratively through the multimodal data video semantics, audio semantics and timeline information of the video content. When the data of a certain modality changes, the associated updates of other modalities are triggered.

[0057] The second aspect of the present invention provides an automatic annotation and acquisition standardization system based on video content, including a video content processing module, a key frame extraction and semantic generation module, an audio processing and multimodal fusion module, a user behavior acquisition and analysis module, and an index database management module;

[0058] The video content processing module is used to decode the original video file into a video frame sequence, adjust the video frames according to the preset resolution and frame rate, and perform standardization processing;

[0059] The keyframe extraction and semantic generation module is used to divide the standardized video frame sequence into multiple time periods according to the time sequence of the frames, calculate the degree of content change between adjacent video frames, select the frame with the largest content change as the keyframe, and generate video semantic information for each keyframe, including spatiotemporal identifiers and preliminary labels. The spatiotemporal identifiers are used to record the time position and segment range of the keyframe, and the preliminary labels are used to record the scene type, object, and action of the keyframe;

[0060] The audio processing and multimodal fusion module is used to extract the audio portion from the original video file, generate audio data synchronized with the video frame time, preprocess the extracted audio data, convert the preprocessed audio data into text using speech recognition, extract the semantic information in the audio, generate corresponding text features, associate and fuse the text features with the spatiotemporal identification and preliminary labels of the key frames, and generate multimodal annotations containing video semantics, audio semantics, and timeline information;

[0061] The user behavior collection and analysis module is used to collect real-time user behavior data during the user's interaction with video content based on multimodal annotation, associate the tags clicked by the user with the corresponding spatiotemporal identifiers, analyze the user's interests and preferences, and dynamically update the keyframe extraction criteria based on the analysis results, including adjusting the threshold of the degree of content change, optimizing the tag classification rules, and adjusting the weight ratio of multiple modalities to optimize the automatic annotation and collection standardization process of video content;

[0062] The index database management module is used to integrate dynamically updated keyframes, tags, and multimodal annotation information into the index database. Based on changes in user behavior and video content, the index database's content and structure are locally adjusted in real time. When data changes, the affected data portion is updated. Based on user retrieval and access frequency, the index structure is adjusted, access channels are established for tags and keyframes with access frequencies higher than a set value, the sorting and priority of index fields are adjusted, index items related to user interests are added, and redundant index information and index information with access frequencies lower than a set value are deleted. This ensures that the index database is updated synchronously with video content and user behavior data. Compared with the prior art, the present invention has at least the following beneficial effects:

[0063] (1) The present invention combines the timestamp information of video frames with visual content features, introduces spatiotemporal identifiers on the basis of the traditional key frame extraction method based on content similarity, and selects the most representative frames in each video according to the chronological order and the degree of content change; not only does it ensure that the selection of key frames is more consistent with the temporal nature of the video content, but also through the generation of spatiotemporal identifiers, subsequent analysis can more accurately capture important information in the video; compared with the existing technology, the present invention can extract key frames from the video more accurately, reduce redundant data, and improve the relevance and accuracy of annotation.

[0064] (2) The present invention proposes to combine the spatiotemporal identification and preliminary labels of the video with the semantic information of the audio stream, and adopt a cross-modal fusion algorithm to generate multimodal annotations containing video, audio and timeline information; different from the traditional modal splicing method, the present invention fuses the deep semantic associations between multiple modalities, which not only improves the annotation accuracy, but also can adapt to different video contents and complex audio information, and improves the ability to process complex video content.

[0065] (3) The present invention combines the real-time behavioral data of users in video browsing, tag selection and interaction to achieve dynamic adjustment of key frame extraction, tag generation and multimodal fusion. The dynamic optimization strategy based on user feedback makes the labeling method personalized and intelligent. By analyzing the user's preferences and behaviors in real time, the labeling strategy can be gradually adjusted to better meet the needs of different users, thereby improving the user experience.

[0066] (4) The present invention dynamically adjusts the annotation information and index structure in the database through real-time monitoring of video annotation content and user behavior, avoiding the static database structure in traditional methods, ensuring the timeliness and accuracy of database content, and enabling the annotation data to be automatically updated as the video content and user needs change, thereby improving the level of intelligent management of annotation data. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 This is a workflow diagram of a method for automatic annotation and acquisition standardization based on video content provided in accordance with an embodiment of the present invention;

[0068] Figure 2 This is a system block diagram of a video content-based automatic annotation and acquisition standardization system provided in accordance with an embodiment of the present invention. DETAILED DESCRIPTION

[0069] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. The embodiments described in this application are only part of the embodiments of the present invention, not all of them. Based on the spirit of the present invention, other embodiments obtained by ordinary technicians in this field without making creative efforts are all within the scope of protection of the present invention.

[0070] Embodiment 1 of the present invention provides a method for automatic annotation and acquisition standardization based on video content, comprising the following steps:

[0071] Step 1: decode the video file into a video frame sequence, adjust the video frames according to the preset resolution and frame rate, and perform standardization processing;

[0072] Preferably, step 1 comprises:

[0073] Decoding the original video file into a sequence of individual video frames, treating each frame as an independent data unit for subsequent processing;

[0074] Converting the decoded video frame sequence into a consistent image format, distributing different video frames to different computing nodes through distributed computing, and selecting the optimal encoding standard according to the computing power and network bandwidth of each node, so that all video frames have the same file format and encoding standard;

[0075] Standardizing the resolution of video frames, scaling the video frames through interpolation algorithms to adjust the resolution of all video frames to the same standard; when adjusting the resolution, the aspect ratio of the original video is kept unchanged, and black edges and cropping are used to match the target video frame resolution;

[0076] Standardizing the frame rate of video frames, removing some frames to meet the target frame rate requirement when the frame rate of video frames does not meet the target frame rate and the number of frames does not match the target requirement; if the number of frames does not meet the target requirement, generate intermediate frames through interpolation method.

[0077] Step 2, divide the standardized video frame sequence into multiple time periods according to the time sequence of the frames, calculate the content change degree between adjacent video frames in each time period, select the frame with the largest content change degree as the key frame, and generate video semantic information for each key frame, including time-space identifier and preliminary label; the time-space identifier is used to record the time position and segment range of the key frame; the preliminary label is used to record the scene type, object and action of the key frame;

[0078] Preferably, step 2 comprises:

[0079] Detecting content changes in the standardized video frame sequence, dividing the video frame sequence into multiple time periods according to the timestamp information of the video frames, analyzing the pixel difference and color histogram change between video frames, and according to a preset threshold, selecting frames with content change degree exceeding the threshold in each time period as the most representative frames, and marking them as key frames;

[0080] Generating time-space identifier for each selected key frame, the time-space identifier records the time position of the key frame and the time range of the segment it is in, and establishes the association between the key frame and the global time axis;

[0081] Identifying and classifying objects in the key frame, using bounding box to mark object position and category information, and de-duplicating and filtering the identified objects;

[0082] Identify and label different objects, scenes, and actions in keyframes and integrate them into preliminary labels, including:

[0083] Perform object detection and classification on each frame, mark the object location and category information with a rectangular box, remove duplicates and filter out the objects that best represent the overall features, and generate corresponding object labels;

[0084] Extract color, texture, and shape features from keyframes, identify scene types using a pre-trained scene classification model, label and classify the identified scenes, and generate corresponding scene labels. Integrate scene information from consecutive keyframes to ensure consistency across all annotations.

[0085] Extract human key points, posture information, and motion trajectories from keyframes, identify motion types using a pre-trained motion classification model, label each motion, and attach the motion labels to the corresponding keyframes for semantic analysis and collection.

[0086] The spatiotemporal identifier records the time position of the keyframe and the time range of the corresponding fragment to establish the association between the keyframe and the global timeline; the preliminary label is generated based on the visual content of the keyframe, including scene type, object features and action description, providing preliminary semantic information for the keyframe.

[0087] Preferably, the performing content change detection on the standardized video frame sequence includes:

[0088] By calculating the pixel differences and color histogram changes between adjacent frames, the degree of change in the frame content is determined;

[0089] The video is segmented into different scenes by analyzing the differences between frames, including: setting a frame threshold, calculating the difference between consecutive frames, and considering the current frame as a key frame if the difference exceeds the preset threshold, and comparing the color histograms of consecutive frames. If the difference between the histograms exceeds the threshold, the current frame is considered a key frame.

[0090] Step 3: Extract the audio portion from the original video file to generate audio data synchronized with the video frame time; preprocess the extracted audio data, convert the preprocessed audio data into text using speech recognition, extract the semantic information in the audio, and generate corresponding text features; associate and fuse the text features with the spatiotemporal identifiers and preliminary labels of the key frames to generate a multimodal annotation containing video semantics, audio semantics, and timeline information;

[0091] Preferably, step 3 includes:

[0092] Based on the video content analysis results, separate the audio stream from the video file, extract the audio stream into an independent file, and convert it to a consistent sampling rate and bit depth;

[0093] Apply frequency-domain filtering and time-domain filtering to the audio stream for noise reduction. This includes extracting frequency-domain features through fast Fourier transform and removing background noise using frequency-domain filters. Furthermore, adaptive filtering and delay cancellation algorithms are combined to provide clear speech input for the speech-to-text process.

[0094] Perform speech-to-text processing on the noise-reduced audio, using a pre-trained automatic speech recognition model to convert speech content into text, recognize the speech content, and output text information. Set a recognition confidence threshold and filter out transcripts below the threshold.

[0095] The generated text is processed, including stop word removal, stemming, and synonym merging. The text content is analyzed for sentiment, the emotional information expressed in the speech is identified, and sentiment tags are generated to provide semantic information for the video content.

[0096] The processed text content and sentiment analysis results are combined with the object, scene and action labels of the video. The text information generated by the audio content is associated with the corresponding video frames and key frame time points according to the key frame time positions recorded in the spatiotemporal identifier. Through the integration of multimodal information, a comprehensive label containing objects, scenes, actions, text and emotions is formed to generate multimodal comprehensive annotation information.

[0097] Preferably, performing speech-to-text processing on the audio after noise reduction processing includes:

[0098] The speech signal is decomposed into multiple frequency sub-bands, and speech features are extracted separately in different frequency sub-bands. By introducing a multi-level neural network structure, the features of different frequency sub-bands are fused.

[0099] Step 4: Based on multimodal annotation, during the user's interaction with the video content, real-time user behavior data is collected, the tags clicked by the user are associated with the corresponding spatiotemporal identifiers, and the user's interests and preferences are analyzed. Based on the analysis results, the keyframe extraction criteria are dynamically updated, including adjusting the threshold of the content change degree, optimizing the tag classification rules, and adjusting the weight ratio of multiple modalities to optimize the automatic annotation and collection standardization process of video content;

[0100] Preferably, step 4 includes:

[0101] Based on multimodal annotation, when users interact with video content, we collect user search, tag click and viewing behavior data, record the user's search keywords, number of tags clicked, viewing time and playback frequency, and store them in the user behavior database;

[0102] Based on the tags clicked by the user, each tag is associated with the corresponding spatiotemporal identifier. The spatiotemporal identifier records the time position and segment range of the key frame in the video. By analyzing the tags clicked by the user and the corresponding spatiotemporal identifier, the user's preferences and interests are obtained and the tag weight is calculated;

[0103] Dynamically update keyframe extraction criteria based on analysis results, including adjusting the threshold for content change, optimizing label classification rules, and distinguishing priorities;

[0104] Adjust the weight ratio of multimodal fusion based on priority, and dynamically adjust the parameter settings of the fusion algorithm based on the user's attention to video semantics and audio semantics;

[0105] The updated tag weights are synchronized to the index database and applied in real time to the search and recommendation functions. At the same time, the tag weight change history is retained so that the latest weight data can be used the next time the user queries, optimizing the automatic annotation and collection standardization process of video content.

[0106] Step 5: Integrate the dynamically updated key frames, tags, and multimodal annotation information into the index database, and make local adjustments to the content and structure of the index database in real time according to changes in user behavior and video content. When the data changes, update the affected data part, adjust the index structure according to the user's retrieval and access frequency, establish access channels for tags and key frames with access frequencies higher than the set value, adjust the sorting and priority of index fields, add index items related to user interests, delete redundant index information and index information with access frequencies lower than the set value, so that the index database is updated synchronously with the video content and user behavior data.

[0107] Preferably, step 5 includes:

[0108] Based on the weight adjustment results, the tag weight data and the corresponding video content annotation information are integrated into the index database. The index database includes comprehensive annotation information of objects, scenes, actions, text and emotions in the video, and assigns a weight to each tag;

[0109] Setting up an index structure, including a main index table, a tag association table, a tag association table, an audio index table, and a video frame index table; storing all associated tags and spatiotemporal information in the tag system in a structured form in a database. Each video entry in the database contains a series of index items, including object tags, scene tags, action tags, text tags, and spatiotemporal information associated with the tags;

[0110] Establish an update mechanism for the index database. When data changes, only the affected data is updated, rather than the entire database.

[0111] Adjust the index structure based on the user's search and access frequency, establish access channels for tags and keyframes with access frequencies higher than the set value, and adjust the sorting and priority of index fields;

[0112] Add index items related to user interests and delete redundant index information or index information with access frequency lower than the set value; the set value range is the number of access times of tags or keyframes that ranks in the top 10% to 30% of the total access volume; the time range is the time window used to count user behavior, which is the last 7 to 30 days starting from the current time;

[0113] Synchronously update the index database with video content and user behavior data to achieve real-time local adjustments to meet users' personalized needs.

[0114] Preferably, establishing an update mechanism for the index database includes:

[0115] When a local change in a video file is detected, the index related to the changed part is updated without re-indexing the entire video file;

[0116] Divide video content and index data into multiple independent content blocks, each of which can be independently indexed and updated; update the index in real time through user interactions, including tagging, video comments, and content suggestions;

[0117] Dynamically adjust the index update direction based on users' real-time search and viewing behaviors, giving priority to updating relevant video indexes that users are interested in;

[0118] The index is updated collaboratively through the multimodal data video semantics, audio semantics and timeline information of the video content. When the data of a certain modality changes, the associated updates of other modalities are triggered.

[0119] Embodiment 2 of the present invention provides an automatic annotation and acquisition standardization system based on video content, including a video content processing module, a key frame extraction and semantic generation module, an audio processing and multimodal fusion module, a user behavior acquisition and analysis module, and an index database management module;

[0120] The video content processing module is used to decode the original video file into a video frame sequence, adjust the video frames according to the preset resolution and frame rate, and perform standardization processing;

[0121] The keyframe extraction and semantic generation module is used to divide the standardized video frame sequence into multiple time periods according to the time sequence of the frames, calculate the degree of content change between adjacent video frames, select the frame with the largest content change as the keyframe, and generate video semantic information for each keyframe, including spatiotemporal identifiers and preliminary labels. The spatiotemporal identifiers are used to record the time position and segment range of the keyframe, and the preliminary labels are used to record the scene type, object, and action of the keyframe;

[0122] The audio processing and multimodal fusion module is used to extract the audio portion from the original video file, generate audio data synchronized with the video frame time, preprocess the extracted audio data, convert the preprocessed audio data into text using speech recognition, extract the semantic information in the audio, generate corresponding text features, associate and fuse the text features with the spatiotemporal identification and preliminary labels of the key frames, and generate multimodal annotations containing video semantics, audio semantics, and timeline information;

[0123] The user behavior collection and analysis module is used to collect real-time user behavior data during the user's interaction with video content based on multimodal annotation, associate the tags clicked by the user with the corresponding spatiotemporal identifiers, analyze the user's interests and preferences, and dynamically update the keyframe extraction criteria based on the analysis results, including adjusting the threshold of the degree of content change, optimizing the tag classification rules, and adjusting the weight ratio of multiple modalities to optimize the automatic annotation and collection standardization process of video content;

[0124] The index database management module is used to integrate dynamically updated keyframes, tags, and multimodal annotation information into the index database. Based on changes in user behavior and video content, the index database's content and structure are locally adjusted in real time. When data changes, the affected data portion is updated. Based on user retrieval and access frequency, the index structure is adjusted, access channels are established for tags and keyframes with access frequencies higher than a set value, the sorting and priority of index fields are adjusted, index items related to user interests are added, and redundant index information and index information with access frequencies lower than a set value are deleted. This ensures that the index database is updated synchronously with video content and user behavior data. Compared with the prior art, the present invention has at least the following beneficial effects:

[0125] Preferably, the video content processing module includes a video preprocessing unit, a frame extraction unit, an object recognition unit, an action recognition unit and a text extraction unit, wherein:

[0126] The video content processing module decodes the video file from its original encoding format into a processable frame sequence, ensuring that all videos are converted to a unified format and encoding standard, adjusting the video resolution as needed, and converting high frame rate videos to standard frame rates;

[0127] The frame extraction unit determines the interval for extracting frames based on the dynamic interval of content changes, detects key frames in the video through histogram comparison and frame difference analysis, and divides the video into different scenes by analyzing the differences between frames. Specifically, the unit sets a frame threshold, calculates the difference between consecutive frames, and considers the current frame to be a key frame if the difference exceeds a preset threshold, compares the color histograms of consecutive frames, and considers the current frame to be a key frame if the difference between the histograms exceeds the threshold;

[0128] The object recognition unit uses an object recognition algorithm to perform deep learning calculations on each frame of the image, and performs object detection and classification through a convolutional neural network model to identify objects in the image. After identifying the object, the object recognition algorithm provides the location information of each detected object and marks the location of the object in the image with a rectangular box. Based on the identified object, the object recognition algorithm can provide specific category information of the object, such as person, car, cat, and further process the identified objects, such as deduplication, screening, and association, to ensure the accuracy of the recognition result;

[0129] The scene recognition unit extracts features from video frames, such as color, texture, and shape, through a deep learning algorithm to describe the characteristic information of the scene. Using a trained scene classification model, the extracted features are analyzed and compared to identify the scene type in the video frame, such as indoor, outdoor, mountain, and sea views. The identified scene type is then labeled and classified, and multiple consecutive frames in the video sequence are integrated and analyzed to add richer descriptive information to the video content, thereby improving the accuracy of video search.

[0130] The action recognition unit uses a deep learning algorithm to extract feature sequences from video frames, including key point coordinates, posture information, and action trajectories, to describe the human body's action characteristics. A pre-trained action classification model is used to analyze and compare the extracted feature sequences to identify different human actions in the video, such as running, dancing, and playing ball. The trained action classification model is used to identify and classify the feature sequences to determine the specific action type displayed in each time period in the video.

[0131] The text extraction unit performs text detection on the video frame, uses a text detection algorithm such as EAST to detect possible text areas in the video frame, determines the position and size of the text, and after detecting the text area, uses text recognition technology such as CRNN to identify the text area, converts the text content into recognizable text information, classifies the text information, such as time, place, and product name, and performs time sequence analysis on consecutive frames of text, taking into account the coherence and time sequence of the text to better identify and process continuous text information.

[0132] Preferably, the audio processing and multimodal fusion module includes an audio processing unit, wherein:

[0133] The audio processing unit separates the audio stream from the video file, converts the audio stream into a unified format and sampling rate, uses a noise reduction algorithm to remove background noise and improve the clarity of the voice, uses an echo cancellation algorithm to reduce and eliminate echo interference, identifies the language in the audio, uses automatic speech recognition (ASR) technology to convert the voice into text, uses natural language processing (NLP) technology to understand the meaning of the text, and normalizes the extracted text, such as removing stop words, stemming, and merging synonyms; extracts keywords and phrases from the converted text, analyzes the emotional tendencies in the voice, such as happiness, sadness, and anger, to provide additional contextual information for the video content, extracts audio features such as pitch, rhythm, and volume for classification and recommendation of video content, and ensures that the results of audio processing are synchronized with the timestamps of the video frames to facilitate subsequent multimodal analysis.

[0134] Preferably, the audio processing and multimodal fusion module further includes an object annotation unit, a scene annotation unit, an action annotation unit, and a text annotation unit, wherein:

[0135] The object labeling unit labels the person, car, and cat identified by the object recognition unit and generates corresponding labels;

[0136] The scene annotation unit annotates scene features according to the colors, textures, and shapes identified by the scene recognition unit and generates corresponding scene feature labels, and annotates scene types according to the identified indoor, outdoor, mountain, and sea views and generates corresponding scene type labels;

[0137] The action labeling unit labels human actions based on the key point coordinates, posture information, and action trajectories identified by the action recognition unit, and generates corresponding human action labels. It labels specific human action types based on the identified running, dancing, and playing actions, and generates corresponding action type labels.

[0138] The text annotation unit generates corresponding text annotations for each frame in the video content according to the text information extracted by the text extraction unit, and generates corresponding text tags.

[0139] Preferably, the keyframe extraction and semantic generation module includes a label integration unit, a label deduplication unit, a label classification unit and a label standardization unit, wherein:

[0140] The label integration unit merges synonyms and related words through text similarity calculation and semantic association analysis to form a more unified label;

[0141] The tag deduplication unit uses a deduplication method based on a hash function to identify and delete duplicate content in the tag;

[0142] The tag classification unit classifies tags according to their semantic and content features, dividing them into different categories and tag groups to facilitate subsequent retrieval and recommendation operations;

[0143] The label standardization unit performs standardization processing on the label, including unifying upper and lower case, adjusting word order, processing special characters, etc., to ensure the consistency and readability of the label.

[0144] Preferably, the index building module includes an index creation unit, an index update unit, a retrieval processing unit and a relevance ranking unit, wherein:

[0145] The index creation unit associates the tag information of the video with the video metadata, and stores the tag and key information associated with each video in the index database to provide support for retrieval;

[0146] The index updating unit monitors changes or updates to video content and regularly updates information in the index database, including tag information for newly added videos and modifications to existing video information;

[0147] The retrieval processing unit receives a user's retrieval request, performs a retrieval operation in the index database according to the query conditions of the request, finds video content that matches the query conditions, and returns it to the user;

[0148] The relevance ranking unit ranks the search results according to the matching degree between the videos and the query conditions and other relevance indicators, and displays the most relevant video content in the front to improve user satisfaction.

[0149] The beneficial effects of the present invention include at least:

[0150] (1) The present invention combines the timestamp information of video frames with visual content features, introduces spatiotemporal identifiers on the basis of the traditional key frame extraction method based on content similarity, and selects the most representative frames in each video according to the chronological order and the degree of content change; not only does it ensure that the selection of key frames is more consistent with the temporal nature of the video content, but also through the generation of spatiotemporal identifiers, subsequent analysis can more accurately capture important information in the video; compared with the existing technology, the present invention can extract key frames from the video more accurately, reduce redundant data, and improve the relevance and accuracy of annotation.

[0151] (2) The present invention proposes to combine the spatiotemporal identification and preliminary labels of the video with the semantic information of the audio stream, and adopt a cross-modal fusion algorithm to generate multimodal annotations containing video, audio and timeline information; different from the traditional modal splicing method, the present invention fuses the deep semantic associations between multiple modalities, which not only improves the annotation accuracy, but also can adapt to different video contents and complex audio information, and improves the ability to process complex video content.

[0152] (3) The present invention combines the real-time behavioral data of users in video browsing, tag selection and interaction to achieve dynamic adjustment of key frame extraction, tag generation and multimodal fusion. The dynamic optimization strategy based on user feedback makes the labeling method personalized and intelligent. By analyzing the user's preferences and behaviors in real time, the labeling strategy can be gradually adjusted to better meet the needs of different users, thereby improving the user experience.

[0153] (4) The present invention dynamically adjusts the annotation information and index structure in the database through real-time monitoring of video annotation content and user behavior, avoiding the static database structure in traditional methods, ensuring the timeliness and accuracy of database content, and enabling the annotation data to be automatically updated as the video content and user needs change, thereby improving the level of intelligent management of annotation data.

[0154] The present disclosure may be a system, method and / or computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A method for automatic annotation and acquisition standardization based on video content, characterized in that: The following steps are involved: Step 1: decode the original video file into a video frame sequence, adjust the video frames according to the preset resolution and frame rate, and perform standardization processing; Step 2: The standardized video frame sequence is divided into multiple time periods according to the time sequence of the frames. In each time period, the content change degree between adjacent video frames is calculated. Frames with content change exceeding the threshold are selected as key frames based on a preset threshold. At the same time, video semantic information is generated for each key frame, including a spatiotemporal identifier and a preliminary label. The spatiotemporal identifier is used to record the time position and segment range of the key frame; the preliminary label is used to record the scene type, object, and action of the key frame. Step 3, extracting the audio portion from the original video file to generate audio data synchronized with the video frame time; Preprocess the extracted audio data, convert the preprocessed audio data into text using speech recognition, extract the semantic information in the audio, and generate corresponding text features; associate and fuse the text features with the spatiotemporal identifiers and preliminary labels of the key frames to generate multimodal annotations that include video semantics, audio semantics, and timeline information; Step 4: Based on multimodal annotation, during the user's interaction with the video content, real-time user behavior data is collected, the tags clicked by the user are associated with the corresponding spatiotemporal identifiers, and the user's interests and preferences are analyzed. Based on the analysis results, the keyframe extraction criteria are dynamically updated, including adjusting the threshold of the content change degree, optimizing the tag classification rules, and adjusting the weight ratio of multiple modalities to optimize the automatic annotation and collection standardization process of video content; Step 5: Integrate the dynamically updated key frames, tags, and multimodal annotation information into the index database, and make local adjustments to the content and structure of the index database in real time according to changes in user behavior and video content. When the data changes, update the affected data part, adjust the index structure according to the user's retrieval and access frequency, establish access channels for tags and key frames with access frequencies higher than the set value, adjust the sorting and priority of index fields, add index items related to user interests, delete redundant index information and index information with access frequencies lower than the set value, so that the index database is updated synchronously with the video content and user behavior data.

2. The method for automatic annotation and acquisition standardization based on video content according to claim 1, characterized in that: Step 1 includes: Decode the original video file into a sequence of separate video frames, so that each frame can be processed as an independent data unit; Convert the decoded video frame sequence into a consistent image format. Through distributed computing, assign different video frames to different computing nodes. Select the optimal encoding standard based on the computing power and network bandwidth of each node, so that all video frames have the same file format and encoding standard. Standardize the resolution of video frames and scale them using an interpolation algorithm to adjust the resolution of all frames to the same standard. When adjusting the resolution, the aspect ratio of the original video is maintained, and black borders and cropping are used to match the target video frame resolution. The frame rate of the video frames is standardized. When the frame rate of the video frames does not meet the target frame rate and the number of frames does not match the target requirements, some frames are removed to meet the target frame rate requirements; if the number of frames does not meet the target requirements, intermediate frames are generated through interpolation.

3. The method for automatic annotation and acquisition standardization based on video content according to claim 1, characterized in that: Step 2 includes: Content change detection is performed on standardized video frame sequences. The video frame sequence is divided into multiple time periods based on the timestamp information of the video frames. The pixel differences and color histogram changes between the video frames are analyzed. Based on a preset threshold, frames with content changes exceeding the threshold are screened out in each time period. These frames are regarded as the most representative frames and marked as key frames. Generate a spatiotemporal identifier for each selected keyframe, which records the time position of the keyframe and the time range of the segment it is in, and establishes an association between the keyframe and the global timeline; Identify and classify objects in the image in key frames, use bounding boxes to mark object locations and category information, and perform deduplication and screening on identified objects; Identify and label different objects, scenes, and actions in keyframes and integrate them into preliminary labels, including: Perform object detection and classification on each frame, mark the object location and category information with a rectangular box, remove duplicates and filter out the objects that best represent the overall features, and generate corresponding object labels; Extract color, texture, and shape features from keyframes, identify scene types using a pre-trained scene classification model, label and classify the identified scenes, and generate corresponding scene labels. Integrate scene information from consecutive keyframes to ensure consistency across all annotations. Extract human key points, posture information, and motion trajectories from keyframes, identify motion types using a pre-trained motion classification model, label each motion, and attach the motion labels to the corresponding keyframes for semantic analysis and collection. The spatiotemporal identifier records the time position of the keyframe and the time range of the corresponding fragment to establish the association between the keyframe and the global timeline; the preliminary label is generated based on the visual content of the keyframe, including scene type, object features and action description, providing preliminary semantic information for the keyframe.

4. The method for automatic annotation and acquisition standardization based on video content according to claim 3, characterized in that: The content change detection on the standardized video frame sequence includes: By calculating the pixel differences and color histogram changes between adjacent frames, the degree of change in the frame content is determined; The video is segmented into different scenes by analyzing the differences between frames, including: setting a frame threshold, calculating the difference between consecutive frames, and considering the current frame as a key frame if the difference exceeds the preset threshold, and comparing the color histograms of consecutive frames. If the difference between the histograms exceeds the threshold, the current frame is considered a key frame.

5. The method for automatic annotation and acquisition standardization based on video content according to claim 1, characterized in that: Step 3 includes: Based on the video content analysis results, separate the audio stream from the video file, extract the audio stream into an independent file, and convert it to a consistent sampling rate and bit depth; Apply frequency-domain filtering and time-domain filtering to the audio stream for noise reduction. This includes extracting frequency-domain features through fast Fourier transform and removing background noise using frequency-domain filters. Furthermore, adaptive filtering and delay cancellation algorithms are combined to provide clear speech input for the speech-to-text process. Perform speech-to-text processing on the noise-reduced audio, using a pre-trained automatic speech recognition model to convert speech content into text, recognize the speech content, and output text information. Set a recognition confidence threshold and filter out transcripts below the threshold. The generated text is processed, including stop word removal, stemming, and synonym merging. The text content is analyzed for sentiment, the emotional information expressed in the speech is identified, and sentiment tags are generated to provide semantic information for the video content. The processed text content and sentiment analysis results are combined with the object, scene and action labels of the video. The text information generated by the audio content is associated with the corresponding video frames and key frame time points according to the key frame time positions recorded in the spatiotemporal identifier. Through the integration of multimodal information, a comprehensive label containing objects, scenes, actions, text and emotions is formed to generate multimodal comprehensive annotation information.

6. The method for automatic annotation and acquisition standardization based on video content according to claim 5, characterized in that: Perform speech-to-text processing on the noise-reduced audio, including: The speech signal is decomposed into multiple frequency sub-bands, and speech features are extracted separately in different frequency sub-bands. By introducing a multi-level neural network structure, the features of different frequency sub-bands are fused.

7. The method for automatic annotation and acquisition standardization based on video content according to claim 1, characterized in that: Step 4 includes: Based on multimodal annotation, when users interact with video content, we collect user search, tag click and viewing behavior data, record the user's search keywords, number of tags clicked, viewing time and playback frequency, and store them in the user behavior database; Based on the tags clicked by the user, each tag is associated with the corresponding spatiotemporal identifier. The spatiotemporal identifier records the time position and segment range of the key frame in the video. By analyzing the tags clicked by the user and the corresponding spatiotemporal identifier, the user's preferences and interests are obtained and the tag weight is calculated; Dynamically update keyframe extraction criteria based on analysis results, including adjusting the threshold for content change, optimizing label classification rules, and distinguishing priorities; Adjust the weight ratio of multimodal fusion based on priority, and dynamically adjust the parameter settings of the fusion algorithm based on the user's attention to video semantics and audio semantics; The updated tag weights are synchronized to the index database and applied in real time to the search and recommendation functions. At the same time, the tag weight change history is retained so that the latest weight data can be used the next time the user queries, optimizing the automatic annotation and collection standardization process of video content.

8. The method for automatic annotation and acquisition standardization based on video content according to claim 1, characterized in that: Step 5 includes: Based on the weight adjustment results, the tag weight data and the corresponding video content annotation information are integrated into the index database. The index database includes comprehensive annotation information of objects, scenes, actions, text and emotions in the video, and assigns a weight to each tag; Setting up an index structure, including a main index table, a tag association table, a tag association table, an audio index table, and a video frame index table; storing all associated tags and spatiotemporal information in the tag system in a structured form in a database. Each video entry in the database contains a series of index items, including object tags, scene tags, action tags, text tags, and spatiotemporal information associated with the tags; Establish an update mechanism for the index database. When data changes, only the affected data is updated, rather than the entire database. Adjust the index structure based on the user's search and access frequency, establish access channels for tags and keyframes with access frequencies higher than the set value, and adjust the sorting and priority of index fields; Add index items related to user interests and delete redundant index information or index information with access frequency lower than the set value; the set value range is the number of access times of tags or keyframes that ranks in the top 10% to 30% of the total access volume; the time range is the time window used to count user behavior, which is the last 7 to 30 days starting from the current time; Synchronously update the index database with video content and user behavior data to achieve real-time local adjustments to meet users' personalized needs.

9. The method for automatic annotation and acquisition standardization based on video content according to claim 1, characterized in that: Establish an update mechanism for the index database, including: When a partial change in a video file is detected, the index related to the changed part is updated without re-indexing the entire video file; Divide video content and index data into multiple independent content blocks, each of which can be independently indexed and updated; update the index in real time through user interactions, including tagging, video comments, and content suggestions; Dynamically adjust the index update direction based on users' real-time search and viewing behaviors, giving priority to updating the index of relevant videos that users are interested in; The index is updated collaboratively through the multimodal data video semantics, audio semantics and timeline information of the video content. When the data of a certain modality changes, the associated updates of other modalities are triggered.

10. A video content-based automatic annotation and collection standardization system, comprising a video content processing module, a keyframe extraction and semantic generation module, an audio processing and multimodal fusion module, a user behavior collection and analysis module, and an index database management module; characterized by: The video content processing module is used to decode the original video file into a video frame sequence, adjust the video frames according to the preset resolution and frame rate, and perform standardization processing; The keyframe extraction and semantic generation module is used to divide the standardized video frame sequence into multiple time periods according to the time sequence of the frames, calculate the degree of content change between adjacent video frames, and select frames with content changes exceeding the threshold as keyframes based on a preset threshold. Video semantic information is generated for each keyframe, including spatiotemporal identifiers and preliminary labels. The spatiotemporal identifiers are used to record the time position and segment range of the keyframe, and the preliminary labels are used to record the scene type, object, and action of the keyframe. The audio processing and multimodal fusion module is used to extract the audio portion from the original video file, generate audio data synchronized with the video frame time, preprocess the extracted audio data, convert the preprocessed audio data into text using speech recognition, extract the semantic information in the audio, generate corresponding text features, associate and fuse the text features with the spatiotemporal identification and preliminary labels of the key frames, and generate multimodal annotations containing video semantics, audio semantics, and timeline information; The user behavior collection and analysis module is used to collect real-time user behavior data during the user's interaction with video content based on multimodal annotation, associate the tags clicked by the user with the corresponding spatiotemporal identifiers, analyze the user's interests and preferences, and dynamically update the keyframe extraction criteria based on the analysis results, including adjusting the threshold of the degree of content change, optimizing the tag classification rules, and adjusting the weight ratio of multiple modalities to optimize the automatic annotation and collection standardization process of video content; The index database management module is used to integrate dynamically updated key frames, tags and multimodal annotation information into the index database, and to make local adjustments to the content and structure of the index database in real time according to changes in user behavior and video content. When the data changes, the affected data part is updated, and the index structure is adjusted according to the user's retrieval and access frequency. Access channels are established for tags and key frames with access frequencies higher than the set value, the sorting and priority of index fields are adjusted, index items related to user interests are added, and redundant index information and index information with access frequencies lower than the set value are deleted, so that the index database is updated synchronously with the video content and user behavior data.

Images