Automatic labeling and acquisition standardization method and system based on video content

Abstract

The invention discloses an automatic labeling and acquisition standardization method and system based on video content, and the method comprises the steps: decoding a video file into a processable video frame sequence, carrying out the standardization processing, unifying the format and the coding standard, and adjusting the resolution and the frame rate; performing content analysis on the standardized video frame sequence, extracting key frames, identifying objects, scenes and actions, and generating a video analysis result and a preliminary label; according to a video content analysis result, separating an audio stream from the video to carry out noise reduction processing and voice-to-text analysis, and combining the processed text with the video content analysis result to generate multi-modal comprehensive annotation information; based on the comprehensive labeling information, taking feedback of the user in video retrieval, label clicking and watching behaviors as an adjustment basis of label weights, and updating weight distribution of labels by analyzing user behavior data; labeling information and weight distribution are integrated into an index database, and indexes are updated in real time according to user behaviors and content changes.

Description

Technical Field

[0001] The present invention belongs to the field of computer Internet technology, and more specifically, relates to a method and system for automatic annotation and collection standardization based on video content. Background Art

[0002] With the rapid development of the Internet and social media, the speed of video content generation and accumulation has increased dramatically, resulting in the emergence of massive video data. Due to the complexity and diversity of video content, manual annotation and management of these videos is time-consuming and costly, so there is an urgent need for an automated video annotation and collection method to improve processing efficiency. In particular, without effective annotation, users often face difficulties in information retrieval and low search efficiency when searching for specific content on video platforms.

[0003] Existing video annotation methods mainly rely on traditional manual annotation or generate tags through simple content analysis. These methods often cannot accurately capture the timing of videos, scene changes, and the semantic association between video and audio content, resulting in low annotation quality and efficiency. For video platforms and social media, the lack of efficient annotation and indexing support greatly reduces the effectiveness of recommendation algorithms, making it difficult to accurately analyze and meet user preferences.

[0004] In addition, as the demand for video content review increases, platforms and content providers need a more efficient annotation system to update content in a timely manner and ensure that platform content complies with relevant regulations. Existing annotation systems often lack dynamic adjustment capabilities and cannot provide real-time optimization when content is updated and user behavior changes, affecting platform content management and user experience.

[0005] Therefore, the market urgently needs a method that can effectively combine the visual features, audio semantics and user behavior of the video to achieve efficient annotation, dynamic optimization and intelligent recommendation of video content. Summary of the invention

[0006] In order to solve the deficiencies in the prior art, the present invention provides a method and system for automatic annotation and acquisition standardization based on video content.

[0007] The present invention adopts the following technical solution.

[0008] A first aspect of the present invention provides a method for automatic annotation and acquisition standardization based on video content, comprising the following steps:

[0009] Step 1, decoding the original video file into a video frame sequence, adjusting the video frames according to a preset resolution and frame rate, and performing standardization processing;

[0010] Step 2: divide the standardized video frame sequence into multiple time periods according to the time sequence of the frames. In each time period, calculate the content change degree between adjacent video frames, select the frame with the largest content change degree as the key frame, and generate video semantic information for each key frame, including spatiotemporal identifier and preliminary label; the spatiotemporal identifier is used to record the time position and segment range of the key frame; the preliminary label is used to record the scene type, object and action of the key frame;

[0011] Step 3: extract the audio part from the original video file to generate audio data synchronized with the video frame time; preprocess the extracted audio data, convert the preprocessed audio data into text using speech recognition, extract the semantic information in the audio, and generate corresponding text features; associate and fuse the text features with the spatiotemporal identification and preliminary labels of the key frames to generate multimodal annotations containing video semantics, audio semantics and timeline information;

[0012] Step 4: Based on multimodal annotation, during the interaction between the user and the video content, the user's real-time behavior data is collected, the tags clicked by the user are associated with the corresponding spatiotemporal identifiers, the user's interest preferences are analyzed, and the key frame extraction criteria are dynamically updated according to the analysis results, including adjusting the threshold of the content change degree, optimizing the tag classification rules, and adjusting the weight ratio of the multimodal to optimize the automatic annotation and acquisition standardization process of the video content;

[0013] Step 5: Integrate the dynamically updated key frames, tags, and multimodal annotation information into the index database, and make local adjustments to the content and structure of the index database in real time according to changes in user behavior and video content. When the data changes, update the affected data portion, adjust the index structure according to the user's retrieval and access frequency, establish access channels for tags and key frames with access frequencies higher than the set value, adjust the sorting and priority of index fields, add index items related to user interests, delete redundant index information and index information with access frequencies lower than the set value, so that the index database is updated synchronously with the video content and user behavior data.

[0014] Preferably, step 1 comprises:

[0015] Decode the original video file into a sequence of separate video frames, so that each frame can be processed as an independent data unit;

[0016] Convert the decoded video frame sequence into a consistent image format. Through distributed computing, assign different video frames to different computing nodes. Select the optimal encoding standard based on the computing power and network bandwidth of each node, so that all video frames have the same file format and encoding standard.

[0017] Standardize the resolution of video frames and scale them through interpolation algorithms to adjust the resolution of all video frames to the same standard. When adjusting the resolution, keep the aspect ratio of the original video unchanged and use black borders and cropping to match the target video frame resolution.

[0018] The frame rate of the video frames is standardized. When the frame rate of the video frames does not meet the target frame rate and the number of frames does not match the target requirement, some frames are removed to meet the target frame rate requirements; if the number of frames does not meet the target requirement, intermediate frames are generated by interpolation.

[0019] Preferably, step 2 comprises:

[0020] Content change detection is performed on the standardized video frame sequence. According to the timestamp information of the video frame, the video frame sequence is divided into multiple time periods. The pixel differences and color histogram changes between the video frames are analyzed. According to the preset threshold, the frames with content changes exceeding the threshold are screened out in each time period as the most representative frames and marked as key frames.

[0021] Generate a time-space mark for each selected key frame, which records the time position of the key frame and the time range of the segment in which it is located, and establishes the association between the key frame and the global time axis;

[0022] Identify and classify objects in the image in key frames, use bounding boxes to mark object locations and category information, and remove duplicates and filter identified objects;

[0023] Identify and label different objects, scenes, and actions in keyframes and integrate them into preliminary labels, including:

[0024] Perform object detection and classification on each frame, use rectangular boxes to mark the object position and category information, remove duplicates and filter out the objects that best represent the overall features, and generate corresponding object labels;

[0025] Extract the features of color, texture and shape in the keyframes, identify the scene type through the pre-trained scene classification model, mark and classify the identified scenes, generate corresponding scene labels, and integrate the scene information of consecutive keyframes to make all annotations consistent;

[0026] Extract human key points, posture information, and motion trajectories from key frames, identify motion types through pre-trained motion classification models, label each motion, and attach motion labels to corresponding key frames for semantic analysis and collection;

[0027] The spatiotemporal identifier records the time position of the keyframe and the time range of the corresponding fragment to establish the association between the keyframe and the global timeline; the preliminary label is generated based on the visual content of the keyframe, including scene type, object features and action description, providing preliminary semantic information for the keyframe.

[0028] Preferably, the performing content change detection on the standardized video frame sequence comprises:

[0029] By calculating the pixel differences between adjacent frames and the amount of change in the color histogram, the degree of change in the frame content is determined;

[0030] By analyzing the differences between frames, the video is segmented into different scenes, including: setting a frame threshold, calculating the difference between consecutive frames, if the difference exceeds the preset threshold, the current frame is considered to be a key frame, and comparing the color histograms of consecutive frames. If the difference between the histograms exceeds the threshold, the current frame is considered to be a key frame.

[0031] Preferably, step 3 comprises:

[0032] According to the video content analysis results, separate the audio stream from the video file, extract the audio stream into an independent file, and convert it to a consistent sampling rate and bit depth;

[0033] Apply frequency domain filtering and time domain filtering methods to the audio stream for noise reduction, including: extracting frequency domain features through fast Fourier transform and using frequency domain filters to remove background noise; combining adaptive filters with delay cancellation algorithms to provide clear voice input for the speech-to-text process;

[0034] Perform speech-to-text processing on the denoised audio, use a pre-trained automatic speech recognition model to convert speech content into text, recognize speech content, and output text information, set a recognition confidence threshold, and filter out transcribed texts below the threshold;

[0035] Process the generated text, including removing stop words, stemming and merging synonyms, analyzing the sentiment tendency of the text content, identifying the emotional information expressed in the speech, and generating emotional tags to provide semantic information for the video content;

[0036] The processed text content and sentiment analysis results are combined with the object, scene and action labels of the video. The text information generated by the audio content is associated with the corresponding video frames and key frame time points according to the key frame time positions recorded in the spatiotemporal identifier. Through the integration of multimodal information, comprehensive labels including objects, scenes, actions, texts and emotions are formed to generate multimodal comprehensive annotation information.

[0037] Preferably, performing speech-to-text processing on the audio after noise reduction processing includes:

[0038] The speech signal is decomposed into multiple frequency sub-bands, and speech features are extracted from different frequency sub-bands respectively. By introducing a multi-level neural network structure, the features of different frequency sub-bands are fused.

[0039] Preferably, step 4 comprises:

[0040] Based on multimodal annotation, when users interact with video content, we collect users’ search, tag click and viewing behavior data, record the keywords users searched, the number of tags they clicked, the viewing time and the playback frequency, and store them in the user behavior database;

[0041] According to the tags clicked by the user, each tag is associated with the corresponding spatiotemporal marker. The spatiotemporal marker records the time position and segment range of the key frame in the video. By analyzing the tags clicked by the user and the corresponding spatiotemporal markers, the user's preferences and interests are obtained and the tag weights are calculated;

[0042] Dynamically update keyframe extraction criteria based on analysis results, including adjusting the threshold for content change, optimizing label classification rules, and distinguishing priorities;

[0043] Adjust the weight ratio of multimodal fusion according to the priority, and dynamically adjust the parameter settings of the fusion algorithm according to the user's attention to video semantics and audio semantics;

[0044] The updated tag weights are synchronized to the index database and applied in real time in the search and recommendation functions. At the same time, the tag weight change history is retained so that the latest weight data can be used the next time the user queries, optimizing the automatic annotation and collection standardization process of video content.

[0045] Preferably, step 5 comprises:

[0046] According to the weight adjustment result, the weight data of the tag and the corresponding video content annotation information are integrated into the index database, which includes the comprehensive annotation information of the video objects, scenes, actions, texts and emotions, and assigns a weight to each tag;

[0047] Setting up an index structure, including a main index table, a tag association table, a tag association table, an audio index table, and a video frame index table; storing all associated tags and spatiotemporal information in the tag system in a structured form in a database, where each video entry contains a series of index items in the database, including object tags, scene tags, action tags, text tags, and spatiotemporal information associated with the tags;

[0048] Establish an update mechanism for the index database. When data changes, update the affected data instead of updating the entire database.

[0049] Adjust the index structure according to the user's search and access frequency, establish access channels for tags and keyframes with access frequencies higher than the set value, and adjust the sorting and priority of index fields;

[0050] Add index items related to user interests, and delete redundant index information and index information with access frequency lower than the set value; the set value range is the number of accesses of tags or keyframes that ranks in the top 10% to 30% of the total access volume; the time range is the time window used to count user behavior, which is the last 7 to 30 days starting from the current time;

[0051] Synchronously update the index database with video content and user behavior data to achieve real-time local adjustments to meet users' personalized needs.

[0052] Preferably, establishing an update mechanism for the index database includes:

[0053] When a local change in a video file is detected, the index related to the changed part is updated without re-indexing the entire video file;

[0054] Divide video content and index data into multiple independent content blocks, each of which can be independently indexed and updated; update the index in real time through user interactions, including tagging, video comments, and content suggestions;

[0055] Dynamically adjust the index update direction based on the user's real-time search and viewing behavior, and give priority to updating the relevant video indexes that the user is interested in;

[0056] The index is updated collaboratively through the multimodal data video semantics, audio semantics and timeline information of the video content. When the data of a certain modality changes, the associated updates of other modalities are triggered.

[0057] The second aspect of the present invention provides an automatic annotation and acquisition standardization system based on video content, including a video content processing module, a key frame extraction and semantic generation module, an audio processing and multimodal fusion module, a user behavior acquisition and analysis module, and an index database management module;

[0058] The video content processing module is used to decode the original video file into a video frame sequence, adjust the video frames according to the preset resolution and frame rate, and perform standardization processing;

[0059] The key frame extraction and semantic generation module is used to divide the standardized video frame sequence into multiple time periods according to the time sequence of the frames, calculate the content change degree between adjacent video frames, select the frame with the largest content change degree as the key frame, and generate video semantic information for each key frame, including spatiotemporal identification and preliminary labels. The spatiotemporal identification is used to record the time position and segment range of the key frame, and the preliminary label is used to record the scene type, object and action of the key frame;

[0060] The audio processing and multimodal fusion module is used to extract the audio part from the original video file, generate audio data synchronized with the video frame time, preprocess the extracted audio data, convert the preprocessed audio data into text using speech recognition, extract the semantic information in the audio, generate the corresponding text features, associate and fuse the text features with the spatiotemporal identification and preliminary labels of the key frames, and generate multimodal annotations containing video semantics, audio semantics and timeline information;

[0061] The user behavior collection and analysis module is used to collect real-time user behavior data based on multimodal annotation during the user's interaction with video content, associate the tags clicked by the user with the corresponding spatiotemporal identifiers, analyze the user's interest preferences, and dynamically update the key frame extraction criteria based on the analysis results, including adjusting the threshold of the content change degree, optimizing the tag classification rules, and adjusting the weight ratio of multimodal to optimize the automatic annotation and collection standardization process of video content;

[0062] The index database management module is used to integrate dynamically updated key frames, tags and multimodal annotation information into the index database, and to make partial adjustments to the content and structure of the index database in real time according to changes in user behavior and video content. When data changes, the affected data part is updated, and the index structure is adjusted according to the user's retrieval and access frequency, and access channels are established for tags and key frames with access frequencies higher than the set value, and the sorting and priority of index fields are adjusted, and index items related to user interests are added, and redundant index information and index information with access frequencies lower than the set value are deleted, so that the index database is updated synchronously with the video content and user behavior data. Compared with the prior art, the beneficial effects of the present invention include at least:

[0063] (1) The present invention combines the timestamp information of video frames with visual content features, introduces spatiotemporal identifiers on the basis of the traditional key frame extraction method based on content similarity, and selects the most representative frames in each video according to the chronological order and the degree of content change. This not only ensures that the selection of key frames is more in line with the temporal nature of the video content, but also enables subsequent analysis to more accurately capture important information in the video through the generation of spatiotemporal identifiers. Compared with the prior art, the present invention can more accurately extract key frames from videos, reduce redundant data, and improve the relevance and accuracy of annotations.

[0064] (2) The present invention proposes to combine the spatiotemporal identification and preliminary labels of the video with the semantic information of the audio stream, and adopt a cross-modal fusion algorithm to generate a multimodal annotation containing video, audio and timeline information. Different from the traditional modal splicing method, the present invention fuses the deep semantic associations between multiple modalities, which not only improves the annotation accuracy, but also can adapt to different video contents and complex audio information, thereby improving the ability to process complex video content.

[0065] (3) The present invention combines the real-time behavioral data of users in video browsing, tag selection and interaction to achieve dynamic adjustment of key frame extraction, tag generation and multimodal fusion. The dynamic optimization strategy based on user feedback makes the labeling method personalized and intelligent. By analyzing the user's preferences and behaviors in real time, the labeling strategy can be gradually adjusted to better meet the needs of different users, thereby improving the user experience.

[0066] (4) The present invention monitors the video annotation content and user behavior in real time, dynamically adjusts the annotation information and index structure in the database, avoids the static database structure in traditional methods, ensures the timeliness and accuracy of the database content, and enables the annotation data to be automatically updated as the video content and user needs change, thereby improving the intelligent management level of the annotation data. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a workflow diagram of a method for automatic annotation and acquisition standardization based on video content provided in accordance with an embodiment of the present invention;

[0068] Figure 2 The system block diagram of a video content-based automatic annotation and acquisition standardization system provided in accordance with an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. The embodiments described in this application are only part of the embodiments of the present invention, not all of them. Based on the spirit of the present invention, other embodiments obtained by ordinary technicians in this field without creative work are all within the scope of protection of the present invention.

[0070] Embodiment 1 of the present invention provides a method for automatic annotation and acquisition standardization based on video content, comprising the following steps:

[0071] Step 1: decode the video file into a video frame sequence, adjust the video frames according to a preset resolution and frame rate, and perform standardization processing;

[0072] Preferably, step 1 comprises:

[0073] Decode the original video file into a sequence of separate video frames, so that each frame can be processed as an independent data unit;

[0074] Convert the decoded video frame sequence into a consistent image format. Through distributed computing, assign different video frames to different computing nodes. Select the optimal encoding standard based on the computing power and network bandwidth of each node, so that all video frames have the same file format and encoding standard.

[0075] Standardize the resolution of video frames and scale them through interpolation algorithms to adjust the resolution of all video frames to the same standard. When adjusting the resolution, keep the aspect ratio of the original video unchanged and use black borders and cropping to match the target video frame resolution.

[0076] The frame rate of the video frames is standardized. When the frame rate of the video frames does not meet the target frame rate and the number of frames does not match the target requirement, some frames are removed to meet the target frame rate requirements; if the number of frames does not meet the target requirement, intermediate frames are generated by interpolation.

[0077] Step 2: divide the standardized video frame sequence into multiple time periods according to the time sequence of the frames. In each time period, calculate the content change degree between adjacent video frames, select the frame with the largest content change degree as the key frame, and generate video semantic information for each key frame, including spatiotemporal identifier and preliminary label; the spatiotemporal identifier is used to record the time position and segment range of the key frame; the preliminary label is used to record the scene type, object and action of the key frame;

[0078] Preferably, step 2 comprises:

[0079] Content change detection is performed on the standardized video frame sequence. According to the timestamp information of the video frame, the video frame sequence is divided into multiple time periods. The pixel differences and color histogram changes between the video frames are analyzed. According to the preset threshold, the frames with content changes exceeding the threshold are screened out in each time period as the most representative frames and marked as key frames.

[0080] Generate a time-space mark for each selected key frame, which records the time position of the key frame and the time range of the segment in which it is located, and establishes the association between the key frame and the global time axis;

[0081] Identify and classify objects in the image in key frames, use bounding boxes to mark object locations and category information, and remove duplicates and filter identified objects;

[0082] Identify and label different objects, scenes, and actions in keyframes and integrate them into preliminary labels, including:

[0083] Perform object detection and classification on each frame, use rectangular boxes to mark the object position and category information, remove duplicates and filter out the objects that best represent the overall features, and generate corresponding object labels;

[0084] Extract the features of color, texture and shape in the keyframes, identify the scene type through the pre-trained scene classification model, mark and classify the identified scenes, generate corresponding scene labels, and integrate the scene information of consecutive keyframes to make all annotations consistent;

[0085] Extract human key points, posture information, and motion trajectories from key frames, identify motion types through pre-trained motion classification models, label each motion, and attach motion labels to corresponding key frames for semantic analysis and collection;

[0086] The spatiotemporal identifier records the time position of the keyframe and the time range of the corresponding fragment to establish the association between the keyframe and the global timeline; the preliminary label is generated based on the visual content of the keyframe, including scene type, object features and action description, providing preliminary semantic information for the keyframe.

[0087] Preferably, the performing content change detection on the standardized video frame sequence comprises:

[0088] By calculating the pixel differences between adjacent frames and the amount of change in the color histogram, the degree of change in the frame content is determined;

[0089] By analyzing the differences between frames, the video is segmented into different scenes, including: setting a frame threshold, calculating the difference between consecutive frames, if the difference exceeds the preset threshold, the current frame is considered to be a key frame, and comparing the color histograms of consecutive frames. If the difference between the histograms exceeds the threshold, the current frame is considered to be a key frame.

[0090] Step 3: extract the audio part from the original video file to generate audio data synchronized with the video frame time; preprocess the extracted audio data, convert the preprocessed audio data into text using speech recognition, extract the semantic information in the audio, and generate corresponding text features; associate and fuse the text features with the spatiotemporal identification and preliminary labels of the key frames to generate multimodal annotations containing video semantics, audio semantics and timeline information;

[0091] Preferably, step 3 comprises:

[0092] According to the video content analysis results, separate the audio stream from the video file, extract the audio stream into an independent file, and convert it to a consistent sampling rate and bit depth;

[0093] Apply frequency domain filtering and time domain filtering methods to the audio stream for noise reduction, including: extracting frequency domain features through fast Fourier transform and using frequency domain filters to remove background noise; combining adaptive filters with delay cancellation algorithms to provide clear voice input for the speech-to-text process;

[0094] Perform speech-to-text processing on the denoised audio, use a pre-trained automatic speech recognition model to convert speech content into text, recognize speech content, and output text information, set a recognition confidence threshold, and filter out transcribed texts below the threshold;

[0095] Process the generated text, including removing stop words, stemming and merging synonyms, analyzing the sentiment tendency of the text content, identifying the emotional information expressed in the speech, and generating emotional tags to provide semantic information for the video content;

[0096] The processed text content and sentiment analysis results are combined with the object, scene and action labels of the video. The text information generated by the audio content is associated with the corresponding video frames and key frame time points according to the key frame time positions recorded in the spatiotemporal identifier. Through the integration of multimodal information, comprehensive labels including objects, scenes, actions, texts and emotions are formed to generate multimodal comprehensive annotation information.

[0097] Preferably, performing speech-to-text processing on the audio after noise reduction processing includes:

[0098] The speech signal is decomposed into multiple frequency sub-bands, and speech features are extracted from different frequency sub-bands respectively. By introducing a multi-level neural network structure, the features of different frequency sub-bands are fused.

[0099] Step 4: Based on multimodal annotation, during the interaction between the user and the video content, the user's real-time behavior data is collected, the tags clicked by the user are associated with the corresponding spatiotemporal identifiers, the user's interest preferences are analyzed, and the key frame extraction criteria are dynamically updated according to the analysis results, including adjusting the threshold of the content change degree, optimizing the tag classification rules, and adjusting the weight ratio of the multimodal to optimize the automatic annotation and acquisition standardization process of the video content;

[0100] Preferably, step 4 comprises:

[0101] Based on multimodal annotation, when users interact with video content, we collect users’ search, tag click and viewing behavior data, record the keywords users searched, the number of tags they clicked, the viewing time and the playback frequency, and store them in the user behavior database;

[0102] According to the tags clicked by the user, each tag is associated with the corresponding spatiotemporal marker. The spatiotemporal marker records the time position and segment range of the key frame in the video. By analyzing the tags clicked by the user and the corresponding spatiotemporal markers, the user's preferences and interests are obtained and the tag weights are calculated;

[0103] Dynamically update keyframe extraction criteria based on analysis results, including adjusting the threshold for content change, optimizing label classification rules, and distinguishing priorities;

[0104] Adjust the weight ratio of multimodal fusion according to the priority, and dynamically adjust the parameter settings of the fusion algorithm according to the user's attention to video semantics and audio semantics;

[0105] The updated tag weights are synchronized to the index database and applied in real time in the search and recommendation functions. At the same time, the tag weight change history is retained so that the latest weight data can be used the next time the user queries, optimizing the automatic annotation and collection standardization process of video content.

[0106] Step 5: Integrate the dynamically updated key frames, tags, and multimodal annotation information into the index database, and make local adjustments to the content and structure of the index database in real time according to changes in user behavior and video content. When the data changes, update the affected data portion, adjust the index structure according to the user's retrieval and access frequency, establish access channels for tags and key frames with access frequencies higher than the set value, adjust the sorting and priority of index fields, add index items related to user interests, delete redundant index information and index information with access frequencies lower than the set value, so that the index database is updated synchronously with the video content and user behavior data.

[0107] Preferably, step 5 comprises:

[0108] According to the weight adjustment result, the weight data of the tag and the corresponding video content annotation information are integrated into the index database, which includes the comprehensive annotation information of the video objects, scenes, actions, texts and emotions, and assigns a weight to each tag;

[0109] Setting up an index structure, including a main index table, a tag association table, a tag association table, an audio index table, and a video frame index table; storing all associated tags and spatiotemporal information in the tag system in a structured form in a database, where each video entry contains a series of index items in the database, including object tags, scene tags, action tags, text tags, and spatiotemporal information associated with the tags;

[0110] Establish an update mechanism for the index database. When data changes, update the affected data instead of updating the entire database.

[0111] Adjust the index structure according to the user's search and access frequency, establish access channels for tags and keyframes with access frequencies higher than the set value, and adjust the sorting and priority of index fields;

[0112] Add index items related to user interests, and delete redundant index information and index information with access frequency lower than the set value; the set value range is the number of accesses of tags or keyframes that ranks in the top 10% to 30% of the total access volume; the time range is the time window used to count user behavior, which is the last 7 to 30 days starting from the current time;

[0113] Synchronously update the index database with video content and user behavior data to achieve real-time local adjustments to meet users' personalized needs.

[0114] Preferably, establishing an update mechanism for the index database includes:

[0115] When a local change in a video file is detected, the index related to the changed part is updated without re-indexing the entire video file;

[0116] Divide video content and index data into multiple independent content blocks, each of which can be independently indexed and updated; update the index in real time through user interactions, including tagging, video comments, and content suggestions;

[0117] Dynamically adjust the index update direction based on the user's real-time search and viewing behavior, and give priority to updating the relevant video indexes that the user is interested in;

[0118] The index is updated collaboratively through the multimodal data video semantics, audio semantics and timeline information of the video content. When the data of a certain modality changes, the associated updates of other modalities are triggered.

[0119] Embodiment 2 of the present invention provides an automatic annotation and collection standardization system based on video content, including a video content processing module, a key frame extraction and semantic generation module, an audio processing and multimodal fusion module, a user behavior collection and analysis module, and an index database management module;

[0120] The video content processing module is used to decode the original video file into a video frame sequence, adjust the video frames according to the preset resolution and frame rate, and perform standardization processing;

[0121] The key frame extraction and semantic generation module is used to divide the standardized video frame sequence into multiple time periods according to the time sequence of the frames, calculate the content change degree between adjacent video frames, select the frame with the largest content change degree as the key frame, and generate video semantic information for each key frame, including spatiotemporal identification and preliminary labels. The spatiotemporal identification is used to record the time position and segment range of the key frame, and the preliminary label is used to record the scene type, object and action of the key frame;

[0122] The audio processing and multimodal fusion module is used to extract the audio part from the original video file, generate audio data synchronized with the video frame time, preprocess the extracted audio data, convert the preprocessed audio data into text using speech recognition, extract the semantic information in the audio, generate the corresponding text features, associate and fuse the text features with the spatiotemporal identification and preliminary labels of the key frames, and generate multimodal annotations containing video semantics, audio semantics and timeline information;

[0123] The user behavior collection and analysis module is used to collect real-time user behavior data based on multimodal annotation during the user's interaction with video content, associate the tags clicked by the user with the corresponding spatiotemporal identifiers, analyze the user's interest preferences, and dynamically update the key frame extraction criteria based on the analysis results, including adjusting the threshold of the content change degree, optimizing the tag classification rules, and adjusting the weight ratio of multimodal to optimize the automatic annotation and collection standardization process of video content;

[0124] The index database management module is used to integrate dynamically updated key frames, tags and multimodal annotation information into the index database, and to make partial adjustments to the content and structure of the index database in real time according to changes in user behavior and video content. When data changes, the affected data part is updated, and the index structure is adjusted according to the user's retrieval and access frequency, and access channels are established for tags and key frames with access frequencies higher than the set value, and the sorting and priority of index fields are adjusted, and index items related to user interests are added, and redundant index information and index information with access frequencies lower than the set value are deleted, so that the index database is updated synchronously with the video content and user behavior data. Compared with the prior art, the beneficial effects of the present invention include at least:

[0125] Preferably, the video content processing module includes a video preprocessing unit, a frame extraction unit, an object recognition unit, an action recognition unit and a text extraction unit, wherein:

[0126] The video content processing module decodes the video file from its original encoding format into a processable frame sequence, ensures that all videos are converted to a unified format and encoding standard, adjusts the video resolution as needed, and converts high frame rate videos to standard frame rates;

[0127] The frame extraction unit determines the interval of extracted frames based on the dynamic interval of content change, detects key frames in the video by histogram comparison and frame difference analysis, and divides the video into different scenes by analyzing the difference between frames, specifically: setting a frame threshold, calculating the difference between consecutive frames, if the difference exceeds a preset threshold, the current frame is considered to be a key frame, comparing the color histograms of consecutive frames, if the difference between the histograms exceeds the threshold, the current frame is considered to be a key frame;

[0128] The object recognition unit uses an object recognition algorithm to perform deep learning calculations on each frame of the image, and performs object detection and classification through a convolutional neural network model to identify the objects in the image. After identifying the objects, the object recognition algorithm gives the position information of each detected object, and marks the position of the object in the image through a rectangular frame. On the basis of identifying the object, the object recognition algorithm can give the specific category information of the object, such as a person, a car, or a cat, and further process the identified objects, such as deduplication, screening, and association, to ensure the accuracy of the recognition results;

[0129] The scene recognition unit extracts features such as color, texture, and shape from video frames through a deep learning algorithm to describe feature information of the scene, and uses a trained scene classification model to analyze and compare the extracted features to identify scene types in the video frames, such as indoor, outdoor, mountain view, and sea view. The identified scene types are marked and classified, and multiple consecutive frames in the video sequence are integrated and analyzed to add richer description information to the video content, thereby improving the accuracy of video search;

[0130] The action recognition unit extracts feature sequences from video frames through a deep learning algorithm, including key point coordinates, posture information, and action trajectories, which are used to describe the action characteristics of the human body. A pre-trained action classification model is used to analyze and compare the extracted feature sequences to identify different human actions in the video, such as running, dancing, and playing ball. The trained action classification model is used to identify and classify the feature sequences to determine the specific action type displayed in each time period in the video.

[0131] The text extraction unit performs text detection on the video frame, uses a text detection algorithm such as EAST to detect possible text areas in the video frame, determines the position and size of the text, and after detecting the text area, uses a text recognition technology such as CRNN to identify the text area, converts the text content into recognizable text information, and classifies the text information, such as time, place, and product name, and performs time sequence analysis on consecutive frames of text, taking into account the coherence and timing of the text, so as to better identify and process continuous text information.

[0132] Preferably, the audio processing and multimodal fusion module comprises an audio processing unit, wherein:

[0133] The audio processing unit separates the audio stream from the video file, converts the audio stream into a unified format and sampling rate, uses a noise reduction algorithm to remove background noise and improve the clarity of speech, uses an echo cancellation algorithm to reduce and eliminate echo interference, identifies the language in the audio, uses automatic speech recognition (ASR) technology to convert speech into text, uses natural language processing (NLP) technology to understand the meaning of the text, and normalizes the extracted text, such as removing stop words, stemming, and merging synonyms; extracts keywords and phrases from the converted text, analyzes the emotional tendencies in the speech, such as happiness, sadness, and anger, to provide additional contextual information for the video content, extracts audio features, such as pitch, rhythm, and volume, for classification and recommendation of video content, and ensures that the results of audio processing are synchronized with the timestamps of video frames to facilitate subsequent multimodal analysis.

[0134] Preferably, the audio processing and multimodal fusion module further includes an object annotation unit, a scene annotation unit, an action annotation unit, and a text annotation unit, wherein:

[0135] The object labeling unit labels the person, car, and cat identified in the object recognition unit and generates corresponding labels;

[0136] The scene annotation unit annotates scene features according to the colors, textures, and shapes identified in the scene recognition unit, and generates corresponding scene feature labels, and annotates scene types according to the identified indoor, outdoor, mountain, and sea views, and generates corresponding scene type labels;

[0137] The action labeling unit labels human actions according to the key point coordinates, posture information, and action trajectories identified in the action recognition unit, and generates corresponding human action labels, and labels specific human action types according to the identified running, dancing, and playing actions, and generates corresponding action type labels;

[0138] The text annotation unit generates corresponding text annotations for each frame in the video content according to the text information extracted by the text extraction unit, and generates corresponding text tags.

[0139] Preferably, the key frame extraction and semantic generation module includes a label integration unit, a label deduplication unit, a label classification unit and a label standardization unit, wherein:

[0140] The label integration unit combines synonyms and related words through text similarity calculation and semantic association analysis to form a more unified label;

[0141] The label deduplication unit uses a deduplication method based on a hash function to identify and delete duplicate content in the label;

[0142] The tag classification unit classifies tags according to their semantic and content features, and divides them into different categories and tag groups to facilitate subsequent retrieval and recommendation operations;

[0143] The label standardization unit performs standardization processing on the label, including unification of upper and lower case, adjustment of word order, special character processing, etc., to ensure the consistency and readability of the label.

[0144] Preferably, the index building module includes an index creation unit, an index update unit, a retrieval processing unit and a relevance ranking unit, wherein:

[0145] The index creation unit associates the tag information of the video with the video metadata, and stores the tag and key information associated with each video in the index database to provide support for retrieval;

[0146] The index updating unit monitors changes or updates to video content and regularly updates information in the index database, including tag information of newly added videos and modifications to existing video information;

[0147] The retrieval processing unit receives a retrieval request from a user, performs a retrieval operation in the index database according to the query conditions of the request, finds video content matching the query conditions, and returns the video content to the user;

[0148] The relevance ranking unit ranks the search results according to the matching degree between the videos in the search results and the query conditions and other relevance indicators, and displays the most relevant video content in the front to improve user satisfaction.

[0149] The beneficial effects of the present invention include at least:

[0150] (1) The present invention combines the timestamp information of video frames with visual content features, introduces spatiotemporal identifiers on the basis of the traditional key frame extraction method based on content similarity, and selects the most representative frames in each video according to the chronological order and the degree of content change. This not only ensures that the selection of key frames is more in line with the temporal nature of the video content, but also enables subsequent analysis to more accurately capture important information in the video through the generation of spatiotemporal identifiers. Compared with the prior art, the present invention can more accurately extract key frames from videos, reduce redundant data, and improve the relevance and accuracy of annotations.

[0151] (2) The present invention proposes to combine the spatiotemporal identification and preliminary labels of the video with the semantic information of the audio stream, and adopt a cross-modal fusion algorithm to generate a multimodal annotation containing video, audio and timeline information. Different from the traditional modal splicing method, the present invention fuses the deep semantic associations between multiple modalities, which not only improves the annotation accuracy, but also can adapt to different video contents and complex audio information, thereby improving the ability to process complex video content.

[0152] (3) The present invention combines the real-time behavioral data of users in video browsing, tag selection and interaction to achieve dynamic adjustment of key frame extraction, tag generation and multimodal fusion. The dynamic optimization strategy based on user feedback makes the labeling method personalized and intelligent. By analyzing the user's preferences and behaviors in real time, the labeling strategy can be gradually adjusted to better meet the needs of different users, thereby improving the user experience.

[0153] (4) The present invention monitors the video annotation content and user behavior in real time, dynamically adjusts the annotation information and index structure in the database, avoids the static database structure in traditional methods, ensures the timeliness and accuracy of the database content, and enables the annotation data to be automatically updated as the video content and user needs change, thereby improving the intelligent management level of the annotation data.

[0154] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents, and any modifications or equivalent replacements that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for automatic annotation and acquisition standardization based on video content, characterized in that: The following steps are involved: Step 1, decoding the original video file into a video frame sequence, adjusting the video frames according to a preset resolution and frame rate, and performing standardization processing; Step 2: divide the standardized video frame sequence into multiple time periods according to the time sequence of the frames. In each time period, calculate the content change degree between adjacent video frames, select the frame with the largest content change degree as the key frame, and generate video semantic information for each key frame, including spatiotemporal identifier and preliminary label; the spatiotemporal identifier is used to record the time position and segment range of the key frame; the preliminary label is used to record the scene type, object and action of the key frame; Step 3, extracting the audio part from the original video file to generate audio data synchronized with the video frame time; Preprocess the extracted audio data, convert the preprocessed audio data into text using speech recognition, extract the semantic information in the audio, and generate corresponding text features; associate and fuse the text features with the spatiotemporal identifiers and preliminary labels of the key frames to generate multimodal annotations containing video semantics, audio semantics, and timeline information; Step 4: Based on multimodal annotation, during the interaction between the user and the video content, the user's real-time behavior data is collected, the tags clicked by the user are associated with the corresponding spatiotemporal identifiers, the user's interest preferences are analyzed, and the key frame extraction criteria are dynamically updated according to the analysis results, including adjusting the threshold of the content change degree, optimizing the tag classification rules, and adjusting the weight ratio of the multimodal to optimize the automatic annotation and acquisition standardization process of the video content; Step 5: Integrate the dynamically updated key frames, tags, and multimodal annotation information into the index database, and make local adjustments to the content and structure of the index database in real time according to changes in user behavior and video content. When the data changes, update the affected data portion, adjust the index structure according to the user's retrieval and access frequency, establish access channels for tags and key frames with access frequencies higher than the set value, adjust the sorting and priority of index fields, add index items related to user interests, delete redundant index information and index information with access frequencies lower than the set value, so that the index database is updated synchronously with the video content and user behavior data.

2. The method for automatic annotation and acquisition standardization based on video content according to claim 1, characterized in that: Step 1 includes: Decode the original video file into a sequence of separate video frames, so that each frame can be processed as an independent data unit; Convert the decoded video frame sequence into a consistent image format. Through distributed computing, assign different video frames to different computing nodes. Select the optimal encoding standard based on the computing power and network bandwidth of each node, so that all video frames have the same file format and encoding standard. Standardize the resolution of video frames and scale them through interpolation algorithms to adjust the resolution of all video frames to the same standard. When adjusting the resolution, keep the aspect ratio of the original video unchanged and use black borders and cropping to match the target video frame resolution. The frame rate of the video frames is standardized. When the frame rate of the video frames does not meet the target frame rate and the number of frames does not match the target requirement, some frames are removed to meet the target frame rate requirements; if the number of frames does not meet the target requirement, intermediate frames are generated by interpolation.

3. The method for automatic annotation and acquisition standardization based on video content according to claim 1, characterized in that: Step 2 includes: Content change detection is performed on the standardized video frame sequence. According to the timestamp information of the video frame, the video frame sequence is divided into multiple time periods. The pixel differences and color histogram changes between the video frames are analyzed. According to the preset threshold, the frames with content changes exceeding the threshold are screened out in each time period as the most representative frames and marked as key frames. Generate a time-space mark for each selected key frame, which records the time position of the key frame and the time range of the segment in which it is located, and establishes the association between the key frame and the global time axis; Identify and classify objects in the image in key frames, use bounding boxes to mark object locations and category information, and remove duplicates and filter identified objects; Identify and label different objects, scenes, and actions in keyframes and integrate them into preliminary labels, including: Perform object detection and classification on each frame, use rectangular boxes to mark the object position and category information, remove duplicates and filter out the objects that best represent the overall features, and generate corresponding object labels; Extract the features of color, texture and shape in the keyframes, identify the scene type through the pre-trained scene classification model, mark and classify the identified scenes, generate corresponding scene labels, and integrate the scene information of consecutive keyframes to make all annotations consistent; Extract human key points, posture information, and motion trajectories from key frames, identify motion types through pre-trained motion classification models, label each motion, and attach motion labels to corresponding key frames for semantic analysis and collection; The spatiotemporal identifier records the time position of the keyframe and the time range of the corresponding fragment to establish the association between the keyframe and the global timeline; the preliminary label is generated based on the visual content of the keyframe, including scene type, object features and action description, providing preliminary semantic information for the keyframe.

4. The method for automatic annotation and acquisition standardization based on video content according to claim 3, characterized in that: The content change detection on the standardized video frame sequence includes: By calculating the pixel differences between adjacent frames and the amount of change in the color histogram, the degree of change in the frame content is determined; By analyzing the differences between frames, the video is segmented into different scenes, including: setting a frame threshold, calculating the difference between consecutive frames, if the difference exceeds the preset threshold, the current frame is considered to be a key frame, and comparing the color histograms of consecutive frames. If the difference between the histograms exceeds the threshold, the current frame is considered to be a key frame.

5. The method for automatic annotation and acquisition standardization based on video content according to claim 1, characterized in that: Step 3 includes: According to the video content analysis results, separate the audio stream from the video file, extract the audio stream into an independent file, and convert it to a consistent sampling rate and bit depth; Apply frequency domain filtering and time domain filtering methods to the audio stream for noise reduction, including: extracting frequency domain features through fast Fourier transform and using frequency domain filters to remove background noise; combining adaptive filters with delay cancellation algorithms to provide clear voice input for the speech-to-text process; Perform speech-to-text processing on the denoised audio, use a pre-trained automatic speech recognition model to convert speech content into text, recognize speech content, and output text information, set a recognition confidence threshold, and filter out transcribed texts below the threshold; Process the generated text, including removing stop words, stemming and merging synonyms, analyzing the sentiment tendency of the text content, identifying the emotional information expressed in the speech, and generating emotional tags to provide semantic information for the video content; The processed text content and sentiment analysis results are combined with the object, scene and action labels of the video. The text information generated by the audio content is associated with the corresponding video frames and key frame time points according to the key frame time positions recorded in the spatiotemporal identifier. Through the integration of multimodal information, comprehensive labels including objects, scenes, actions, texts and emotions are formed to generate multimodal comprehensive annotation information.

6. The method for automatic annotation and acquisition standardization based on video content according to claim 5, characterized in that: Perform speech-to-text processing on the noise-reduced audio, including: The speech signal is decomposed into multiple frequency sub-bands, and speech features are extracted from different frequency sub-bands respectively. By introducing a multi-level neural network structure, the features of different frequency sub-bands are fused.

7. The method for automatic annotation and acquisition standardization based on video content according to claim 1, characterized in that: Step 4 includes: Based on multimodal annotation, when users interact with video content, we collect users’ search, tag click and viewing behavior data, record the keywords users searched, the number of tags they clicked, the viewing time and the playback frequency, and store them in the user behavior database; According to the tags clicked by the user, each tag is associated with the corresponding spatiotemporal marker. The spatiotemporal marker records the time position and segment range of the key frame in the video. By analyzing the tags clicked by the user and the corresponding spatiotemporal markers, the user's preferences and interests are obtained and the tag weights are calculated; Dynamically update keyframe extraction criteria based on analysis results, including adjusting the threshold for content change, optimizing label classification rules, and distinguishing priorities; Adjust the weight ratio of multimodal fusion according to the priority, and dynamically adjust the parameter settings of the fusion algorithm according to the user's attention to video semantics and audio semantics; The updated tag weights are synchronized to the index database and applied in real time in the search and recommendation functions. At the same time, the tag weight change history is retained so that the latest weight data can be used the next time the user queries, optimizing the automatic annotation and collection standardization process of video content.

8. The method of automatic annotation and acquisition standardization based on video content according to claim 1, characterized in that: Step 5 includes: According to the weight adjustment result, the weight data of the tag and the corresponding video content annotation information are integrated into the index database, which includes the comprehensive annotation information of the video objects, scenes, actions, texts and emotions, and assigns a weight to each tag; Setting up an index structure, including a main index table, a tag association table, a tag association table, an audio index table, and a video frame index table; storing all associated tags and spatiotemporal information in the tag system in a structured form in a database, where each video entry contains a series of index items in the database, including object tags, scene tags, action tags, text tags, and spatiotemporal information associated with the tags; Establish an update mechanism for the index database. When data changes, update the affected data instead of updating the entire database. Adjust the index structure according to the user's search and access frequency, establish access channels for tags and keyframes with access frequencies higher than the set value, and adjust the sorting and priority of index fields; Add index items related to user interests, and delete redundant index information and index information with access frequency lower than the set value; the set value range is the number of accesses of tags or keyframes that ranks in the top 10% to 30% of the total access volume; the time range is the time window used to count user behavior, which is the last 7 to 30 days starting from the current time; Synchronously update the index database with video content and user behavior data to achieve real-time local adjustments to meet users' personalized needs.

9. The method of automatic annotation and acquisition standardization based on video content according to claim 1, characterized in that: Establish an update mechanism for the index database, including: When a local change in a video file is detected, the index related to the changed part is updated without re-indexing the entire video file; Divide video content and index data into multiple independent content blocks, each of which can be independently indexed and updated; update the index in real time through user interactions, including tagging, video comments, and content suggestions; Dynamically adjust the index update direction based on the user's real-time search and viewing behavior, and give priority to updating the relevant video indexes that the user is interested in; The index is updated collaboratively through the multimodal data video semantics, audio semantics and timeline information of the video content. When the data of a certain modality changes, the associated updates of other modalities are triggered.

10. An automatic annotation and collection standardization system based on video content, comprising a video content processing module, a key frame extraction and semantic generation module, an audio processing and multimodal fusion module, a user behavior collection and analysis module and an index database management module; characterized in that: The video content processing module is used to decode the original video file into a video frame sequence, adjust the video frames according to the preset resolution and frame rate, and perform standardization processing; The key frame extraction and semantic generation module is used to divide the standardized video frame sequence into multiple time periods according to the time sequence of the frames, calculate the content change degree between adjacent video frames, select the frame with the largest content change degree as the key frame, and generate video semantic information for each key frame, including spatiotemporal identification and preliminary labels. The spatiotemporal identification is used to record the time position and segment range of the key frame, and the preliminary label is used to record the scene type, object and action of the key frame; The audio processing and multimodal fusion module is used to extract the audio part from the original video file, generate audio data synchronized with the video frame time, preprocess the extracted audio data, convert the preprocessed audio data into text using speech recognition, extract the semantic information in the audio, generate the corresponding text features, associate and fuse the text features with the spatiotemporal identification and preliminary labels of the key frames, and generate multimodal annotations containing video semantics, audio semantics and timeline information; The user behavior collection and analysis module is used to collect real-time user behavior data based on multimodal annotation during the user's interaction with video content, associate the tags clicked by the user with the corresponding spatiotemporal identifiers, analyze the user's interest preferences, and dynamically update the key frame extraction criteria based on the analysis results, including adjusting the threshold of the content change degree, optimizing the tag classification rules, and adjusting the weight ratio of multimodal to optimize the automatic annotation and collection standardization process of video content; The index database management module is used to integrate dynamically updated key frames, tags and multimodal annotation information into the index database, and to make local adjustments to the content and structure of the index database in real time according to changes in user behavior and video content. When the data changes, the affected data part is updated, and the index structure is adjusted according to the user's retrieval and access frequency. Access channels are established for tags and key frames with access frequencies higher than set values, the sorting and priority of index fields are adjusted, index items related to user interests are added, and redundant index information and index information with access frequencies lower than set values ​​are deleted, so that the index database is updated synchronously with video content and user behavior data.

Images