A video tag classification method, system and product based on metadata extraction and intelligent scene division

Abstract

The application relates to the field of video processing, and discloses a video label classification method, system and product based on metadata extraction and intelligent scene division, which comprises the following steps: S1, acquiring an input video and converting the input video into an intermediate format code; S2, separating a video image track, an audio track and a subtitle track; S3, extracting metadata of the video, specifically comprising the following steps: S301, extracting image metadata, comparing the similarity of video intervals based on the image metadata, thereby dividing a video scene and generating a scene visual label; S302, extracting scene audio metadata according to the divided scene, thereby generating a scene auditory label; S303, extracting text metadata according to the divided scene, thereby generating a scene text label; and S4, based on different scene labels, combining the weight components and time points of each scene in the video, and comprehensively obtaining a video label. The application can comprehensively extract and summarize metadata of a video, widen the dimension of a video label, and improve the accuracy of video recognition and matching.

Description

Technical Field

[0001] This application relates to the field of video processing technology, specifically to a video tag classification method, system, and product based on metadata extraction and intelligent scene segmentation. Background Technology

[0002] Traditional methods for video content analysis and video tagging primarily involve identifying image content in video images based on manual annotation or computer vision (CV) algorithms, summarizing it into structured information, and using this as the basis for video analysis. However, these methods have significant shortcomings:

[0003] Manual processing is costly and inefficient, and is not suitable for large-scale video data processing.

[0004] Traditional video content extraction methods have limited information dimensions and cannot deeply understand the abstract semantics of videos, such as style, emotion, and story theme. They also lack the ability to integrate and analyze multimodal content, including images, sound, and subtitles, resulting in inconsistent information quality and making it difficult to guarantee the stability of video tagging. In particular, long videos are significantly more difficult to tag than short videos due to their rich content and much greater information volume. Summary of the Invention

[0005] The purpose of this application is to overcome the shortcomings of the prior art and provide a video tag classification method, system and product based on metadata extraction and intelligent scene segmentation. It can realize metadata extraction and information summarization and recognition for videos of various lengths, expand the dimensions of video tagging, and improve the confidence of video recognition and tagging.

[0006] Firstly, this application provides a video tag classification method based on metadata extraction and intelligent scene segmentation, the technical solution of which includes the following steps:

[0007] S1: Acquire and transcode the input video, converting it into an intermediate format encoding for storage;

[0008] S2 separates the video into image tracks, audio tracks, and subtitle tracks;

[0009] S3 extracts the video's metadata, specifically including:

[0010] S301, extract image metadata from the image track, obtain the similarity of different video intervals based on the image metadata, divide the video into scenes based on the similarity, and generate scene visual labels;

[0011] S302, extract the audio metadata of the audio tracks of each scene according to the divided scenes, and generate scene auditory tags;

[0012] S303, extract the text metadata of the subtitle track for each scene according to the divided scenes, and generate scene text tags;

[0013] S4, based on the scene labels of different scenarios, combines the weight components of each scene in the video and the deviation of the time of appearance from the timeline to obtain the video's labels.

[0014] The above technical solution divides the video into three dimensions: image track, audio track, and subtitle track. Image metadata, audio metadata, and text metadata are extracted respectively, enabling comprehensive extraction of video information from multiple dimensions. By dividing the video into different scenes based on image metadata, adding scene tags according to different scenes, and then synthesizing video tags through scene tags, the generalization of video tags can be avoided, enabling precise tag annotation of the video. It also enables refined definition and recommendation of video content, facilitating accurate location of target content in the video.

[0015] Preferably, the intermediate format encoding is H.264 / AVC format encoding, and the length of the Group of Pictures (GOP) encoded in H.264 / AVC format is set according to the length of the video.

[0016] By employing the above technical solution, and using H.264 / AVC format encoding with adjustable GOP length as the intermediate format encoding for video metadata extraction, the length of the GOP can be adapted to the video length, thereby improving the adaptability of subsequent video metadata extraction and scene segmentation.

[0017] Preferably, step S301 involves extracting image metadata from the image track, obtaining similarity scores for different video intervals based on the image metadata, segmenting the video scenes based on the similarity scores, and generating scene visual labels. Specifically, this includes:

[0018] S3011, Based on the intermediate format encoding, obtain a continuously arranged group of images (GOP), and extract image metadata using the group of images (GOP) as the object;

[0019] S3012, a sliding window with a preset width and interval is used to scan the image track of the video to obtain continuously arranged video intervals. Any video interval is composed of several image groups (GOPs), and the image metadata of each image group (GOP) forms the image metadata matrix of the video interval.

[0020] S3013, calculate the similarity of the image metadata matrix between adjacent video intervals, and make a similarity judgment. If the similarity is lower than the threshold, it is determined that a scene switch has occurred, and the first image group GOP in the subsequent video interval is defined as the time point of the video scene switch; S3014, add scene visual labels to the scene based on the image metadata for different scenes.

[0021] By employing the aforementioned technical solution, and using video intervals composed of consecutive image groups (GOPs) obtained through sliding window scanning, similarity comparison is performed on a per-interval basis. This avoids the problems of inaccurate scene segmentation and overly dense scene segmentation caused by camera switching when using single-frame comparison for scene segmentation. Furthermore, by performing similarity comparison based on the metadata matrix of video intervals formed by extracting video metadata from image groups (GOPs), the video metadata extracted from image tracks can be fully and comprehensively utilized to comprehensively determine the nodes for scene segmentation, thereby improving the accuracy of scene segmentation.

[0022] Preferably, the extraction of image metadata based on a Group of Pictures (GOP) specifically includes:

[0023] A dual-channel U-Net network is used to distinguish the foreground and background of a group of images (GOPs).

[0024] Feature extraction is performed on the background, extracting metadata such as edge pixel density, background texture, and perspective relationship of the background, and then performing feature fusion to generate background metadata;

[0025] The foreground is distinguished and identified as people and objects. Features are extracted from people and objects respectively to obtain image metadata and object metadata.

[0026] Feature extraction is performed on the color data of the image to obtain image color metadata;

[0027] For different scenarios, scene visual tags are added to the scene based on image metadata, specifically including:

[0028] An environment label is generated based on the background metadata, a person label is obtained based on the portrait metadata, a scene description label is obtained based on the object metadata, and a visual emotion classification label is obtained based on the image color metadata.

[0029] The specific method for obtaining visual emotion classification labels based on image color metadata is as follows:

[0030] Feature extraction is performed on the color data of the image to obtain the hue H. c Saturation S c Brightness L c and pixel density D s The visual emotion score E is calculated from the image color metadata.s As shown in the following formula,

[0031] E s =γ1·H c +γ2·S c +γ3·L c +γ4·D s

[0032] Where γ1, γ2, γ3, and γ4 are adjustable weight parameters, the calculation result yields a unique solution E. s Visual Emotion Score E s This is mapped to visual emotion classification labels, which are Calm, Tense, Joyful, Melancholy, and Intense.

[0033] The above technical solution extracts image metadata of different dimensions based on image group GOP, which serves as the basis for comparing the similarity of video intervals, and obtains the corresponding scene visual labels based on the image metadata of different dimensions.

[0034] Preferably, step S302 involves extracting audio metadata from the audio tracks according to the divided scenes and generating scene auditory tags, specifically including:

[0035] S3021, Based on the divided scenes, the audio tracks are segmented and the audio tracks of each scene are extracted;

[0036] S3022 separates background noise and vocals from the audio track;

[0037] S3023, perform feature recognition on background noise to obtain environmental sound labels;

[0038] S3024, extract features from human voice to obtain auditory emotion classification labels;

[0039] The specific method for extracting features from human voices to obtain voice emotion classification labels is as follows:

[0040] Feature extraction is performed on the human voice to obtain the pitch. std and intensity Energy mean Human voice metadata is used to calculate the auditory emotion score E. h As shown in the following formula,

[0041] E h =α·Pitch std +β·Energy mean

[0042] Where α and β are training parameters, the calculation result yields a unique solution E. hAuditory emotion score E h This is mapped to auditory emotion classification labels, which are Calm, Tense, Joyful, Melancholy, and Intense.

[0043] The above technical solution divides the audio track based on scene segmentation, extracts audio metadata of different dimensions of the audio track according to the scene, and obtains the corresponding scene auditory tags based on the audio metadata of different dimensions.

[0044] Preferably, step S303 involves extracting the text metadata of the subtitle track for each scene according to the divided scenes, and generating scene text tags, specifically including:

[0045] S3031, Extract the text data from the subtitle track and segment the text data according to the divided scenes;

[0046] S3032, extract word information from the text data to obtain independent word information w. i A semantic redundancy fusion algorithm is used to obtain the information w of each word. i Fusion score S(w) i The fusion scoring function is as follows:

[0047] S(w i )=λ1·f(w i )+λ2·p(w i )+λ3·TF-IDF(w i )

[0048] Where, f(w) i ) represents word information w i Word frequency component score, p(w i ) represents word information w i Location-weighted component scoring, TF-IDF(w i ) represents word information w i The specific weighted component scores are given, where λ1, λ2, and λ3 are adjustable weight parameters.

[0049] S3033, Screening fusion score S(w) i The top-ranked words in the sorting information i As text metadata for the subtitle track, scene text tags are obtained based on the text metadata.

[0050] The above technical solution divides the subtitle track based on scene segmentation, extracts the text metadata of the subtitle track according to the scene, filters out the core content in the corresponding scene text metadata through a fusion scoring function, and obtains the corresponding scene text tags.

[0051] Preferably, in step S4, based on the scene tags of different scenes, and combining the weight components of each scene in the video and the deviation of its occurrence time from the timeline, the video tags are obtained comprehensively. The specific method is as follows:

[0052] Based on the time allotted to each scene in the total video duration, weight coefficients are added to each tag of each scene. The scene with the longest duration is designated as the main scene, and the deviations of the other scenes from the main scene are used to adjust the weight coefficients. The weighted sum of the same tags of the same type in each scene is calculated to obtain the weight score of the scene tag. The top-ranked scene tags are then output as video tags.

[0053] Preferably, step S4 further includes adding narrative type tags and creative style tags to the video by using the visual emotion tags or the auditory emotion classification tags at different distribution locations and frequencies in the video.

[0054] Through the above technical solutions, the accuracy and interpretability of video tag generation can be improved by focusing on spatiotemporal weights and anchoring the main scene. This enables video tags to reflect the core theme of the video, eliminate noise tags, and provide information dimensions such as narrative type tags and creative style tags for the video.

[0055] Secondly, this application provides a system for implementing a video tag classification method based on metadata extraction and intelligent scene segmentation, the specific technical solution of which includes:

[0056] The system includes a video input module, a multimodal parsing module, a metadata extraction module, a metadata structured storage module, and an interface output module. The video input module receives and manages the input video, converts it into an intermediate format encoding, and stores it.

[0057] The multimodal parsing module separates the video into image tracks, audio tracks, and subtitle tracks;

[0058] The metadata extraction module includes a visual analysis submodule, an auditory analysis submodule, a text analysis submodule, and a fusion generation submodule;

[0059] The visual analysis submodule extracts image metadata from the image track, divides the video into scenes based on the image metadata, and generates scene visual tags.

[0060] The auditory analysis submodule extracts the audio metadata of the audio tracks of each scene according to the divided scenes, and generates scene auditory tags.

[0061] The text analysis submodule extracts the text metadata of the subtitle track for each scene according to the divided scenes, and generates scene text tags;

[0062] The fusion generation submodule, based on the scene tags of different scenes, combines the weight components of each scene in the video and the deviation of the time of appearance from the timeline to obtain the video tags.

[0063] The metadata structured storage module saves the video tag results output by the metadata extraction module as a structured video information file, which is convenient for subsequent external system calls or for external search engine indexing.

[0064] The interface output module provides a standard API interface to push structured video information files to third-party content management systems, recommendation systems, or search engines, or to export them as external files for storage or archiving.

[0065] Thirdly, this application provides a computer program product, which includes a computer program or instructions that enable the computer program or instructions to implement the steps of the video tag classification method based on metadata extraction and intelligent scene segmentation described above.

[0066] In summary, this application includes at least one of the following beneficial technical effects:

[0067] 1. This application, by separating the video track, audio track, and subtitle track and extracting metadata from each, can identify multi-dimensional content elements of the video, including not only static images but also sound, subtitles, and semantic content. It can obtain a deeper understanding of video content information, identify and define video elements including emotions and themes, expand the dimensions of video tags, thereby improving the accuracy of video search, association, and push, and enhancing the user experience.

[0068] 2. This application compares the similarity of video intervals based on image metadata to achieve accurate video scene segmentation. In particular, it can distinguish scenes and label long videos, which can solve the semantic generalization problem caused by labeling objects based on the entire video. It refines video speech capture and label definition, and retains long-tail label information. Users can jump to specific scenes through specific labels, improving the efficiency of obtaining information while watching videos.

[0069] 3. This application has a high degree of automation in processing input video, which greatly reduces the degree of human intervention and can significantly improve the efficiency of video information processing.

[0070] 4. The multidimensional metadata obtained in this application is ultimately output as a formatted video tag system, which is convenient for retrieval and retrieval, highly adaptable, and can be connected to third-party content management platforms or media asset systems. Attached Figure Description

[0071] Figure 1This is a flowchart illustrating a video tag classification method based on metadata extraction and intelligent scene segmentation according to Embodiment 1 of this application;

[0072] Figure 2 This is a flowchart illustrating step S3 in Embodiment 1 of this application;

[0073] Figure 3 This is a flowchart illustrating step S301 in Embodiment 1 of this application;

[0074] Figure 4 This is a schematic diagram illustrating the use of a sliding window for video interval similarity comparison in Embodiment 1 of this application;

[0075] Figure 5 This is a flowchart illustrating step S302 in Embodiment 1 of this application;

[0076] Figure 6 This is a flowchart illustrating step S303 in Embodiment 1 of this application;

[0077] Figure 7 This is a schematic diagram of the architecture of the video tag classification system in Embodiment 2 of this application. Detailed Implementation

[0078] This specific embodiment is merely an explanation of this application and is not intended to limit it. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they are within the scope of this application.

[0079] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. It should be noted that in the optional embodiments of this application, the object information and other related data involved require the permission or consent of the object when the embodiments of this application are applied to specific products or technologies, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. That is to say, if the embodiments of this application involve data related to the object, it needs to be obtained with the authorization and consent of the object, the authorization and consent of the relevant departments, and in compliance with the relevant laws, regulations, and standards of the country and region. If personal information is involved in the embodiments, the acquisition of all personal information requires the consent of the individual. If sensitive information is involved, the separate consent of the information subject is required, and the embodiments also need to be implemented with the authorization and consent of the object.

[0080] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0081] Example 1:

[0082] Please see Figure 1 This application discloses a video tag classification method based on metadata extraction and intelligent scene segmentation, which includes the following steps:

[0083] S1 acquires and transcodes the input video, converting it into an intermediate format encoding of H.264 / AVC for storage. The length of the Group of Pictures (GOP) encoded in H.264 / AVC format is set according to the length of the video.

[0084] A Group of Pictures (GOP) consists of I-frames, P-frames, and B-frames arranged in a specific order. The I-frame is the keyframe, and the P-frames and B-frames use the I-frame as a reference frame. A GOP consists of one I-frame and several B / P-frames. The length of a GOP is the number of frames between the I-frames of one GOP and the I-frames of the next GOP. The length of a GOP encoded using H.264 / AVC format can be adjusted according to parameters, ensuring that the GOP length matches the video length during transcoding. Longer videos use longer GOPs, which improves compression ratio, reduces bitrate fluctuations caused by I-frame insertion, ensures video continuity, and reduces the cost of storing video for subsequent feature extraction calculations. Shorter videos use shorter GOPs because they often contain faster editing and transitions, with more frequent scene changes. Shorter GOPs ensure accurate perception of high-frequency shots, improving the completeness of video information acquisition.

[0085] S2 separates the video into image tracks, audio tracks, and subtitle tracks.

[0086] After the video has been transcoded into an intermediate format, the image track, audio track, and subtitle track of the video are extracted separately so that different dimensions of video metadata can be extracted according to different tracks.

[0087] S3 extracts different modal metadata from the image track, audio track, and subtitle track, and adds different dimensional tags to the video based on the video's metadata.

[0088] It should be noted that while video metadata itself can serve as information representing video content for storage, transmission, and recognition, cross-modal integration issues arise in practical applications due to the different dimensions of different modal metadata. Therefore, by adding tags of different dimensions to videos based on their metadata, cross-modal semantic alignment of different modal metadata can be achieved, improving the uniformity and comprehensibility of video semantic expression, and facilitating subsequent implementations such as video retrieval and recommendation using video tags.

[0089] For details, please refer to Figure 2 Extracting video metadata specifically includes:

[0090] S301, extract image metadata from the image track. The image metadata has two functions: first, to obtain the similarity between different video intervals based on the image metadata, and to divide the video into scenes based on the similarity; second, to generate scene visual labels for each scene based on the image metadata.

[0091] For more details, please see Figure 3 The specific implementation methods for the above steps include:

[0092] S3011, Based on the intermediate format encoding, obtain a continuously arranged group of images (GOP), and extract image metadata using the group of images (GOP) as the object.

[0093] Since the video has been transcoded to an intermediate encoding format suitable for its length, the length of a Group of Pictures (GOP) is already determined. A single GOP serves as the basic unit for extracting image metadata, resulting in extracted image metadata. This image metadata is a collection of multimodal image metadata from which various types of features and semantic meanings are extracted from the GOP.

[0094] It should be noted that a Group of Pictures (GOP) is a frame sequence consisting of one I-frame and several B / P frames, representing a video frame of a certain duration. The I-frame is a full-frame compressed coded frame that encodes and transmits the entire frame's image information. During decoding, the complete image can be reconstructed using only the I-frame data, describing details of the background and moving subjects, without needing to refer to other frames. The P-frame uses the I-frame as a reference frame, finding the predicted values ​​and motion vectors of the P-frame within the I-frame, and encoding and transmitting the prediction difference and motion vectors. During decoding, the predicted values ​​and prediction errors in the I-frame must be summed to reconstruct the complete P-frame image. The B-frame uses the preceding I-frame or P-frame and the following P-frame as reference frames, and is a bidirectional predictive coded frame.

[0095] When extracting image metadata from a Group of Pictures (GOP), information is primarily obtained by extracting the images from I-frames and combining this information with the motion vectors and prediction residuals from P-frames and B-frames to obtain multidimensional image metadata. The specific methods for extracting multidimensional image metadata will be explained in detail later.

[0096] S3012 uses a sliding window with preset width and interval to scan the video image track and obtain continuously arranged video intervals. Any video interval is composed of several image groups (GOPs), and the image metadata of each GOP forms the image metadata matrix of the video interval.

[0097] Because the specific content described in the video over time and the timing of scene transitions are unknown, the span of a Group of Pictures (GOP) may cover scene content information with significant differences before and after scene transitions. If a single keyframe or a single GOP is used as the basis for comparing the similarity of scene transitions, large errors may occur.

[0098] Please see Figure 4 In this application, a technical solution is adopted to scan the image track using a sliding window with a preset width and interval. One step of the sliding window is one video interval. The width of the sliding window refers to the number of image groups (GOPs) it comprises, i.e., the number of GOPs the video interval consists of. The interval refers to the number of GOPs the sliding window moves in each step. Taking this application's embodiment as an example, the preset width of the sliding window is 3 GOPs, and the interval is 3 GOPs. When the interval equals the width, the video interval can cover the entire image track end-to-end. It should be noted that the interval does not necessarily have to be equal to the width of the sliding window; it can be adjusted as needed to allow for partial overlap between video intervals.

[0099] By using video intervals as the similarity comparison object for scene transitions, we can obtain all video metadata of an image group (GOP) within a certain time span, obtain a video metadata matrix, and use the video metadata matrix as the object for similarity comparison. This can ensure the semantic integrity and content coherence of video intervals, guarantee that the sliding window covers the time boundary of scene transitions, reduce the semantic expression difficulty of scene transition detection, and improve the accuracy of scene transition detection.

[0100] S3013, calculate the similarity of the image metadata matrices between adjacent video intervals, perform similarity judgment, if the similarity is lower than the threshold, determine that a scene change has occurred, and define the first image group (GOP) in the subsequent video interval as the time point of the video scene change. Mark the timeline position of the I-frame of this image group GOP to provide anchor points for scene segmentation in subsequent data processing.

[0101] S3014 adds scene visual tags to scenes based on image metadata, corresponding to different scenarios.

[0102] More specifically, the above steps involve extracting multidimensional image metadata using Group of Pictures (GOPs) as the object, specifically including:

[0103] A dual-channel U-Net network is used to distinguish the foreground and background of a group of images (GOP).

[0104] Specifically, in the dual-channel U-Net network, the RGB image of the I-frame of the group of images (GOP) is input into the spatial feature channel to extract static appearance features; the inter-frame difference map generated from the P / B frames is input into the temporal feature channel to quantize motion vectors. Through adaptive feature fusion using an attention mechanism, the final output includes the background static region, foreground target (person, object), and transition boundary region between the foreground and background of the GOP.

[0105] Background features are extracted, including edge pixel density, background texture, and perspective metadata. These are then fused to generate background metadata. The edge pixel density reflects the structural complexity of the background and can be obtained by generating a binary edge map using the Canny algorithm. Background texture is described by the spatial distribution of pixel grayscale values, used to distinguish different materials or surfaces, and is obtained from the grayscale co-occurrence matrix. Perspective is used to quantify scene depth information, inferring the 3D spatial layout by analyzing geometric structures. The above technical solution is implemented using existing algorithms and will not be elaborated upon here. By recognizing the background metadata, environmental labels for the scene can be obtained, such as indoor environments, urban street environments, forest environments, beach environments, and space environments.

[0106] The system distinguishes between people and objects in the foreground using a deep convolutional neural network. Features are extracted from both to obtain image metadata and object metadata. Image metadata includes facial feature metadata and styling feature metadata, used to anchor images appearing in the video. The identity and number of images can be used as criteria for scene transitions. By recognizing image metadata, person tags are obtained, including gender, age, occupation, and clothing. For specific images, a celebrity database can be used to anchor specific celebrity identities, adding corresponding celebrity tags to the video's scene. Object metadata obtains scene description tags by recognizing object features. For example, recognizing an operating table and a shadowless lamp yields an operating room tag, recognizing a hospital bed and an IV stand yields a ward tag, recognizing a bookshelf and books yields a library tag, and recognizing a desk and a blackboard yields a classroom tag.

[0107] Feature extraction is performed on the video color data to obtain video color metadata, and visual emotion classification labels are obtained based on the video color metadata. Specifically, a self-developed model, the Style-ToneScaler (STS) model, is used to classify the emotional tendency of the video's color statistical features.

[0108] Feature extraction is performed on the color data of the image to obtain the hue H. c Saturation S c Brightness L c and pixel density D s The visual emotion score E is calculated from the image color metadata. s As shown in the following formula,

[0109] E s =γ1·H c +γ2·S c +γ3·L c +γ4·D s

[0110] Where γ1, γ2, γ3, and γ4 are adjustable weight parameters, the calculation result yields a unique solution E. s Visual Emotion Score E s This is mapped to visual emotion classification labels, which are Calm, Tense, Joyful, Melancholy, and Intense.

[0111] After acquiring and storing the multidimensional video metadata of the group of images (GOP), a multidimensional video metadata matrix of the video interval is obtained using the aforementioned sliding window scanning technique, and then a similarity comparison is performed. The specific method for comparing the similarity of the video metadata matrix is ​​existing technology and can be implemented by various different methods. In the embodiments of this application, the Manhattan distance is calculated for multidimensional data matrix similarity comparison.

[0112] Based on scene segmentation, environmental labels are generated based on the background metadata, character labels are obtained based on the portrait metadata, scene description labels are obtained based on the object metadata, and visual emotion classification labels are obtained based on the image color metadata, thereby adding scene visual labels to each different scene of the video segmentation.

[0113] Building upon the existing scene segmentation, we continue by extracting metadata from the audio and subtitle tracks. Specifically, this includes:

[0114] S302: Extract audio metadata from the audio tracks of each scene according to the defined scenes, and generate scene auditory tags. Please refer to [link / reference]. Figure 5 Specifically, it includes:

[0115] S3021, based on the defined scenes, the audio tracks are segmented, and the audio tracks for each scene are extracted.

[0116] S3022, Separate background noise and vocals from the audio track. In embodiments of this application, the NovaVSS tool can be used to extract background noise and vocals separately.

[0117] S3023 performs feature recognition on background noise to obtain environmental sound labels.

[0118] Specifically, in this embodiment, environmental sound tags are divided into different categories, including natural environment, urban environment, indoor environment, and special environment. Natural environment includes forests, beaches, mountains, rainforests, etc., dominated by low-frequency natural sounds with gentle energy changes and uniform spectral distribution; urban environment includes streets, subway stations, shopping malls, construction areas, etc., with a mixture of mid-to-high frequency background human voices / machinery sounds and frequent energy abrupt changes, such as car horns and crowd noise; indoor environment includes offices, restaurants, libraries, and hospitals, with high clarity of background human voices, stable background noise, and a spectrum concentrated in the 300–2000Hz range; special environment includes concerts, stadiums, and airports, with extremely high-energy pulses, such as cheers / engine sounds, and a wide bandwidth covering 20Hz–20kHz. It should be noted that the background human voices described above are different from the human voices separated in S3022.

[0119] S3024, feature extraction is performed on human voice to obtain auditory emotion classification labels. The specific method is as follows:

[0120] Feature extraction is performed on the human voice to obtain the pitch. std and intensity Energy mean Human voice metadata is used to calculate the auditory emotion score E. h As shown in the following formula,

[0121] E h =α·Pitch std +β·Energy mean

[0122] Wherein, α and β are training parameters, which can be fined through repeated testing and human emotion judgment, and the calculation result yields a unique solution E. h Auditory emotion score E h These tags map to auditory emotion classification labels, which are categorized as Calm, Tense, Joyful, Melancholy, and Intense. These auditory emotion classification labels complement and reference the visual emotion classification labels, forming video emotion tags in combination.

[0123] S303: Extract the text metadata of the subtitle track for each scene according to the divided scenes, and generate scene text tags. Please refer to [link / reference]. Figure 6 Specifically, it includes:

[0124] S3031. Extract the text data of the subtitle track and segment the text data according to the divided scenes.

[0125] S3032. Extract word information from the text data to obtain independent word information w i , and use the semantic redundancy fusion algorithm to obtain the fusion score S(w i ) of each word information w i . The specific fusion score function is as follows:

[0126] S(w i ) = λ1·f(w i ) + λ2·p(w i ) + λ3·TF-IDF(w i )

[0127] where f(w i ) is the word frequency component score of the word information w i , obtained according to the frequency of occurrence of the word information w i ; p(w i ) is the position weight component score of the word information w i , assigned according to the position where the word information w i appears in the text data; TF-IDF(w i ) is the specificity weight component score of the word information w i , obtained according to the co-occurrence frequency with other word information, to avoid the word frequency component score f(w i ) being unable to exclude common words including "de" and "shi"; λ1, λ2, and λ3 are adjustable weight parameters.

[0128] S3033. Screen the word information w i with a relatively high ranking of the fusion score S(w i ) as the text metadata of the subtitle track, and use the LLM large language model to identify the scene text label that generally describes the theme content of the video from the text metadata, so as to describe the content scope and vertical field of the video.

[0129] In another embodiment, when performing S2, the step of separating the image track, audio track, and subtitle track of the video, multi-source text metadata is obtained by a supplementary method. Specifically, it includes performing periodic OCR recognition on video frames, and extracting independent subtitles and superimposed text (OSD) in specific areas of the screen as text metadata; using the Whisper pre-trained model to recognize the human voice extracted in S3023 with the Whisper model, and taking the speech-to-text processing result as text metadata.

[0130] S4 automatically generates video tags based on scene tags for different scenarios, combined with the weight of each scene in the video and the deviation of its occurrence time from the timeline, and integrates the video metadata extraction results from all modalities. These tags include dimensions such as content summary, sentiment tags, themes, keywords, and recommendation level.

[0131] Specifically, weight coefficients are added to each tag of each scene according to the time of each scene in the total video duration. The scene with the longest duration is taken as the main scene, and the deviation of the other scenes from the main scene is used as the correction of the weight coefficients. The weight coefficients of the same tags in the same type of each scene are adjusted and weighted to obtain the weight score of the scene tag. The top-ranked scene tags are output as video tags.

[0132] Simply overlaying tags from different scenes can cause multiple secondary tags from different scenes to overwhelm the core tags. By using duration weighting and deviation weighting of the main scene to correct the error, the true proportion of tags in the overall video can be more accurately reflected, thereby improving tag accuracy and user intent matching.

[0133] In another embodiment, the video tags are obtained by combining information such as the distribution location, frequency and distribution method of visual emotion tags and auditory emotion classification tags in the video, and then adding narrative type tags and creative style tags to the video.

[0134] For example, if the "Joyful" emotion tag accounts for more than 60% of the video, and the ending scene is also labeled "Joyful," then a "Comedy" tag is added to the video; if the video contains the "Melancholy" emotion tag, and the ending scene is labeled "Tense," "Melancholy," or "Intense," then a "Tragedy" tag is added to the video; if the emotion tags of the video scenes range from "Tense" and "Intense" to "Calm" and "Joyful," then a "Healing" tag is added to the video; if the visual emotion tag is "Joyful" and the auditory emotion tag is "Melancholy," then a "Contrast Montage" tag is added. The above embodiments are specific applications of adding tags to videos based on emotion tags and scene distribution, and do not constitute a limitation on the technical solution of this application.

[0135] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0136] Example 2:

[0137] Please see Figure 7This application provides a video tag classification system to implement the video tag classification method based on metadata extraction and intelligent scene segmentation, comprising: a video input module 1, a multimodal parsing module 2, a metadata extraction module 3, a metadata structured storage module 4, and an interface output module 5.

[0138] Video input module 1 receives and manages the input video, converts it to H.264 / AVC format encoding for storage, and determines the length of the Group of Pictures (GOP) encoded in H.264 / AVC format based on the video length. Video input module 1 supports various video media sources such as local upload, URL address, and API interface. After downloading and transcoding the video, it is stored in the local processing cache.

[0139] The multimodal parsing module 2 separates the video into image tracks, audio tracks, and subtitle tracks.

[0140] The metadata extraction module 3 is the core module of the system. It uses multiple AI models to extract and analyze features from different modalities to generate video metadata. Specifically, it includes: visual analysis submodule 31, auditory analysis submodule 32, text analysis submodule 33, and fusion generation submodule 34.

[0141] The visual analysis submodule 31 extracts image metadata from the image track, divides the video into scenes based on the image metadata, and generates scene visual tags.

[0142] The auditory analysis submodule 32 extracts the audio metadata of the audio tracks of each scene according to the divided scenes, and generates scene auditory tags.

[0143] The text analysis submodule 33 extracts the text metadata of the subtitle track for each scene according to the divided scenes, and generates scene text tags.

[0144] The fusion generation submodule 34, based on the scene tags of different scenarios, combines the weight components of each scene in the video and the deviation of the time point of occurrence from the timeline to obtain the video tags.

[0145] In another embodiment, the metadata extraction module 3 further includes a structure analysis submodule 35 and an AI plugin submodule 36.

[0146] The structural analysis submodule 35 analyzes the structure of video media files, extracting information such as encoding format, duration, bit rate, resolution, audio channels, etc., which can increase the dimensions of video metadata and video tags.

[0147] AI Plugin Submodule 36: For the various intermediate video metadata extracted above, the configured AI plugin is invoked for in-depth analysis, such as tracking updated vertical domain data or current hot data. The plugin system supports the integration of pre-trained models conforming to the MCP standard interface specification, automatically transmitting target metadata and pre-training configurations. Plugin processing results can be returned via various formats including IPC gateway, shared memory, cache queue, and database, facilitating expansion and secondary development. When vertical domain or hot information is implemented, the plugin system can integrate a proprietary Data-Agent, incorporating local datasets to positively impact the processing results with richer domain data, generating more representative analysis results.

[0148] The metadata structured storage module 4 saves the video tag results output by the metadata extraction module as a structured video information file, supporting standard JSON / RDF or other database structures, which is convenient for subsequent external system calls or for external search engine indexing.

[0149] The interface output module 5 provides a standard API interface, which supports pushing structured video information files to third-party content management systems, recommendation systems or search engines, or exporting them as external files for storage or archiving.

[0150] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the charging pile transaction processing system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0151] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0152] Example 3:

[0153] An embodiment of this application provides a computer program product, including a computer program or instructions that enable the computer program or instructions to implement the steps of the video tag classification method based on metadata extraction and intelligent scene segmentation described above.

[0154] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0155] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A video tag classification method based on metadata extraction and intelligent scene segmentation, characterized in that, Includes the following steps: S1: Acquire and transcode the input video, converting it into an intermediate format encoding for storage; S2 separates the video into image tracks, audio tracks, and subtitle tracks; S3 extracts the video's metadata, specifically including: S301, extract image metadata from the image track, obtain the similarity of different video intervals based on the image metadata, divide the video into scenes based on the similarity, and generate scene visual labels; S302, extract the audio metadata of the audio tracks of each scene according to the divided scenes, and generate scene auditory tags; S303, extract the text metadata of the subtitle track for each scene according to the divided scenes, and generate scene text tags; S4, based on the scene labels of different scenarios, combines the weight components of each scene in the video and the deviation of its occurrence time from the timeline to obtain the video's labels. The specific method is as follows: Based on the time allotted to each scene in the total video duration, weight coefficients are added to each tag of each scene. The scene with the longest duration is designated as the main scene, and the deviations of the other scenes from the main scene are used to adjust the weight coefficients. The weighted sum of the same tags of the same type in each scene is calculated to obtain the weight score of the scene tag. The top-ranked scene tags are then output as video tags.

2. The video tag classification method based on metadata extraction and intelligent scene segmentation according to claim 1, characterized in that, The intermediate format encoding is H.264 / AVC format encoding, and the length of the Group of Pictures (GOP) encoded in H.264 / AVC format is set according to the length of the video.

3. The video tag classification method based on metadata extraction and intelligent scene segmentation according to claim 1, characterized in that, S301 involves extracting image metadata from the image track, obtaining similarity scores for different video intervals based on the image metadata, segmenting the video scenes based on the similarity scores, and generating scene visual labels, specifically including: S3011, Based on the intermediate format encoding, obtain a continuously arranged group of images (GOP), and extract image metadata using the group of images (GOP) as the object; S3012, a sliding window with a preset width and interval is used to scan the image track of the video to obtain continuously arranged video intervals. Any video interval is composed of several image groups (GOPs), and the image metadata of each image group (GOP) forms the image metadata matrix of the video interval. S3013, calculate the similarity of the image metadata matrix between adjacent video intervals, make a similarity judgment, and if the similarity is lower than the threshold, it is determined that a scene switch has occurred, and the first image group GOP in the subsequent video interval is defined as the time point of the video scene switch. S3014 adds scene visual tags to scenes based on image metadata, corresponding to different scenarios.

4. The video tag classification method based on metadata extraction and intelligent scene segmentation according to claim 3, characterized in that, The extraction of image metadata based on Group of Pictures (GOPs) specifically includes: A dual-channel U-Net network is used to distinguish the foreground and background of a group of images (GOPs). Feature extraction is performed on the background, extracting metadata such as edge pixel density, background texture, and perspective relationship of the background, and then performing feature fusion to generate background metadata; The foreground is distinguished and identified as people and objects. Features are extracted from people and objects respectively to obtain image metadata and object metadata. Feature extraction is performed on the color data of the image to obtain image color metadata; For different scenarios, scene visual tags are added to the scene based on image metadata, specifically including: An environment label is generated based on the background metadata, a person label is obtained based on the portrait metadata, a scene description label is obtained based on the object metadata, and a visual emotion classification label is obtained based on the image color metadata. The specific method for obtaining visual emotion classification labels based on image color metadata is as follows: Feature extraction is performed on the color data of the image to obtain the hue H. c Saturation S c Brightness L c and pixel density D s The visual emotion score E is calculated from the image color metadata. s As shown in the following formula, ； in, The weighting parameters are adjustable, and the calculation yields a unique solution E. s Visual Emotion Score E s This is mapped to visual emotion classification labels, which are Calm, Tense, Joyful, Melancholy, and Intense.

5. The video tag classification method based on metadata extraction and intelligent scene segmentation according to claim 1, characterized in that, In step S302, audio metadata of the audio tracks is extracted according to the divided scenes to generate scene auditory tags, specifically including: S3021, Based on the divided scenes, the audio tracks are segmented and the audio tracks of each scene are extracted; S3022 separates background noise and vocals from the audio track; S3023, perform feature recognition on background noise to obtain environmental sound labels; S3024, extract features from human voice to obtain auditory emotion classification labels; The specific method for extracting features from human voices to obtain voice emotion classification labels is as follows: Feature extraction is performed on the human voice to obtain the pitch. std and intensity Energy mean Human voice metadata is used to calculate the auditory emotion score E. h As shown in the following formula, ； Where α and β are training parameters, the calculation result yields a unique solution E. h Auditory emotion score E h This is mapped to auditory emotion classification labels, which are Calm, Tense, Joyful, Melancholy, and Intense.

6. The video tag classification method based on metadata extraction and intelligent scene segmentation according to claim 1, characterized in that, S303 involves extracting the text metadata of the subtitle track for each scene according to the divided scenes, and generating scene text tags, specifically including: S3031, Extract the text data from the subtitle track and segment the text data according to the divided scenes; S3032, extract word information from the text data to obtain independent word information w. i A semantic redundancy fusion algorithm is used to obtain the information w of each word. i Fusion score S(w) i The fusion scoring function is as follows: ； in, For word information w i Word frequency component scoring, For word information w i Position weight component scoring, For word information w i Specific weighted component score, These are adjustable weight parameters; S3033, Screening fusion score S(w) i The top-ranked words in the sorting information i As text metadata for the subtitle track, scene text tags are obtained based on the text metadata.

7. A video tag classification method based on metadata extraction and intelligent scene segmentation according to claim 4, characterized in that, S4, based on the scene tags of different scenes, combined with the weight components of each scene in the video and the deviation of the time of appearance from the timeline, comprehensively obtains the video tags, and also includes adding narrative type tags and creative style tags to the video based on the different distribution positions and frequencies of the visual emotion classification tags in the video.

8. A video tagging and classification system based on metadata extraction and intelligent scene segmentation, characterized in that, include: Video input module, multimodal parsing module, metadata extraction module, metadata structured storage module, and interface output module; The video input module receives and manages the input video, converts it into an intermediate format encoding, and stores it. The multimodal parsing module separates the video into image tracks, audio tracks, and subtitle tracks; The metadata extraction module includes a visual analysis submodule, an auditory analysis submodule, a text analysis submodule, and a fusion generation submodule; The visual analysis submodule extracts image metadata from the image track, divides the video into scenes based on the image metadata, and generates scene visual tags. The auditory analysis submodule extracts the audio metadata of the audio tracks of each scene according to the divided scenes, and generates scene auditory tags. The text analysis submodule extracts the text metadata of the subtitle track for each scene according to the divided scenes, and generates scene text tags; The fusion generation submodule, based on the scene tags of different scenes, combines the weight components of each scene in the video and the deviation of the time of appearance from the timeline to obtain the video tags. The metadata structured storage module saves the video tag results output by the metadata extraction module as a structured video information file, which is convenient for subsequent external system calls or for external search engine indexing. The interface output module provides a standard API interface to push structured video information files to third-party content management systems, recommendation systems, or search engines, or to export them as external files for storage or archiving.

9. A computer program product, characterized in that, The computer program product includes a computer program or instructions that enable the computer program or instructions to implement the steps of the video tag classification method based on metadata extraction and intelligent scene segmentation as described in any one of claims 1 to 7.

Images