Video tag classification method, system and product based on metadata extraction and intelligent scene division

Abstract

The invention relates to the field of video processing, and discloses a video tag classification method, system and product based on metadata extraction and intelligent scene division, and the method comprises the steps: S1, obtaining an input video, and converting the input video into an intermediate format code; s2, separating a video image track, an audio track and a subtitle track; s3, extracting metadata of the video, specifically, S301, extracting image metadata, and performing video interval similarity comparison based on the image metadata so as to divide a video scene and generate a scene visual label; s302, generating a scene auditory label according to the divided scene audio metadata; s303, extracting text metadata according to the divided scene, and generating a scene text label; and S4, based on different scene labels, combining the weight component and the occurrence time point of each scene in the video to comprehensively obtain a video label. According to the method, comprehensive extraction and induction of the metadata of the video can be realized, the video tag dimension is expanded, and the accuracy of video identification matching is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video processing, in particular to a video label classification method, system and product based on metadata extraction and intelligent scene division. BACKGROUND

[0002] The main way of traditional video content analysis and video label annotation is to identify the image content in the image of the video based on manual annotation or CV (computer vision) algorithm, and summarize it as structured information, which is used as the basis for video analysis. The above technical solution has the following significant deficiencies: The cost of manual processing is high and the efficiency is low, which is not suitable for large-scale video data processing; The information dimension extracted by the traditional CV method is limited, which cannot deeply understand the abstract semantic understanding of the video, such as style, emotion, story theme, lacks the fusion analysis ability of multi-modal content including image, sound, subtitle, etc. The quality of the information obtained is uneven, and the stability of the video label annotation cannot be guaranteed; especially for long videos, due to the rich video content, the information quantity is much larger than that of short videos, and the difficulty of label annotation is significantly greater than that of general short videos. SUMMARY

[0003] The purpose of the present application is to overcome the deficiencies of the prior art, provide a video label classification method, system and product based on metadata extraction and intelligent scene division, which can realize metadata extraction and information induction recognition of videos of various lengths, expand the dimension of video label annotation, and improve the confidence of video recognition and annotation.

[0004] In a first aspect, the present application provides a video label classification method based on metadata extraction and intelligent scene division, which adopts the following steps: S1, obtaining and transcoding the input video, and converting it into an intermediate format code for storage; S2, separating the video into image track, audio track and subtitle track; S3, extracting the metadata of the video, specifically including: S301, extracting the image metadata of the image track, obtaining the similarity of different video intervals based on the image metadata, and dividing the scenes of the video based on the similarity to generate scene visual labels; S302, extracting the audio metadata of the audio track of each scene according to the divided scenes to generate scene auditory labels; S303, extracting the text metadata of the subtitle track of each scene according to the divided scenes to generate scene text labels; S4, based on the respective scene labels of different scenes, combining the weight components of respective scenes in the video and the deviation values of the time points of respective scenes relative to the timeline, comprehensively obtaining the label of the video.

[0005] By the above technical solution, the video is divided into three dimensions of image track, audio track and subtitle track, and the image metadata, audio metadata and text metadata are extracted respectively, so that the video information can be comprehensively extracted from multiple dimensions; by dividing the video into different scenes based on the image metadata, adding scene labels according to different scenes, and comprehensively obtaining the video label through the scene labels, the video label generalization can be avoided, the precise label annotation of the video can be realized, and the video content refinement definition and recommendation can be realized, so that the target content position of the video can be accurately positioned.

[0006] Preferably, the intermediate format coding is H.264 / AVC format coding, and the length of the group of pictures GOP of the H.264 / AVC format coding is set according to the length of the video.

[0007] By the above technical solution, by using the H.264 / AVC format coding with adjustable group of pictures GOP length as the intermediate format coding for video metadata extraction, the length of the group of pictures GOP can be adapted to the length of the video, and the adaptation degree of subsequent video metadata extraction and scene division can be improved.

[0008] Preferably, in S301, the image metadata of the image track is extracted, the similarity of different video intervals is obtained based on the image metadata, the scene division of the video is performed based on the similarity, and the scene visual label is generated, specifically comprising: S3011, obtaining the continuously arranged group of pictures GOP based on the intermediate format coding, and extracting the image metadata by taking the group of pictures GOP as an object; S3012, scanning the image track of the video by using a sliding window with a preset width and interval to obtain continuously arranged video intervals, any video interval is composed of a plurality of group of pictures GOP, and the image metadata of each group of pictures GOP constitutes the image metadata matrix of the video interval; S3013, calculating the similarity of the image metadata matrices between adjacent video intervals, performing similarity judgment, and if the similarity is lower than a threshold value, it is judged that a scene switching occurs, and the first group of pictures GOP in the subsequent video interval is defined as the time point of the video scene switching; S3014, adding scene visual labels to scenes based on the image metadata according to different scenes.

[0009] By the technical solution, the video interval formed by the continuous image group GOP obtained through the sliding window scanning is compared in similarity in units of the video interval, which can avoid the problem of inaccurate scene division and too dense scene division caused by the switching of the lens when a single frame picture comparison is used for scene division; meanwhile, the metadata matrix of the video interval formed based on the video metadata extracted from the image group GOP is compared in similarity, which can fully and comprehensively utilize the video metadata extracted from the image track to comprehensively judge the node of scene division and improve the accuracy of scene division.

[0010] As preferred, the image metadata is extracted from the image group GOP, specifically including: The foreground and the background of the image group GOP are distinguished by using a double-channel U-Net network; The background is subjected to feature extraction to extract the metadata of the edge pixel density, background texture and perspective relationship of the background, and the background metadata is generated after feature fusion; The foreground is subjected to portrait and object distinguishing and recognition, and the portrait metadata and the object metadata are obtained through feature extraction of the portrait and the object, respectively; The picture color data is subjected to feature extraction to obtain the picture color metadata; The image metadata is added to the scene to add scene visual labels to the scene based on the image metadata, specifically including: The environment label is generated based on the background metadata, the character label is obtained based on the portrait metadata, the scene description label is obtained based on the object metadata, and the visual emotion classification label is obtained based on the picture color metadata; The specific method for obtaining the visual emotion classification label based on the picture color metadata is: The picture color data is subjected to feature extraction to obtain the hue H c , the saturation S c , the brightness L c and the pixel point density D s of the picture color metadata, and the visual emotion score E s is calculated as shown in the following formula, E s =γ1·H c +γ2·S c +γ3·L c +γ4·D s wherein, γ1, γ2, γ3, γ4 are adjustable weight parameters, and the calculation result obtains a unique solution E s , and the visual emotion score E smapping with visual emotion classification labels, the types of visual emotion classification labels are respectively Calm, Tense, Joyful, Melancholy and Intense.

[0011] By the above technical solution, different dimensional image metadata is extracted based on the image group GOP as the basis for video interval similarity comparison, and corresponding scene visual labels are obtained according to different dimensional image metadata.

[0012] As preferred, the S302, audio metadata of the audio track is extracted according to the divided scene to generate a scene auditory label, specifically including: S3021, the audio track is segmented according to the divided scene to extract the audio track of each scene; S3022, the audio track is separated into background sound and human voice; S3023, the background sound is identified to obtain an environmental sound label; S3024, the human voice is extracted to obtain an auditory emotion classification label; Wherein, the specific method for extracting the human voice to obtain the sound emotion classification label is: The human voice is extracted to obtain the human voice metadata of pitch std and intensity Energy mean , and the auditory emotion score E h is calculated as shown in the following formula, E h = α·Pitch std + β·Energy mean Wherein, α and β are training parameters, and the calculation result obtains a unique solution E h , the auditory emotion score E h forms a mapping with the auditory emotion classification label, and the types of auditory emotion classification labels are respectively Calm, Tense, Joyful, Melancholy and Intense.

[0013] By the above technical solution, the audio track is segmented based on scene division, and different dimensional audio metadata of the audio track is extracted according to the scene, and corresponding scene auditory labels are obtained according to different dimensional audio metadata.

[0014] As preferred, the S303, text metadata of the subtitle track of each scene is extracted according to the divided scene to generate a scene text label, specifically including: S3031, the text data of the subtitle track is extracted, and the text data is segmented according to the divided scene; S3032, text data is extracted for word information, and independent word information w is obtained i , a semantic redundancy fusion algorithm is used to obtain a fusion score S(w i ) of each word information w i , and the fusion score function is specifically as follows: S(w i ) = λ1·f(w i ) + λ2·p(w i ) + λ3·TF-IDF(w i ) wherein f(w i ) is a word frequency component score of the word information w i , p(w i ) is a position weight component score of the word information w i , TF-IDF(w i ) is a specificity weight component score of the word information w i , and λ1, λ2 and λ3 are adjustable weight parameters; S3033, word information w i with a high fusion score S(w i ) is selected as text metadata of a subtitle track, and scene text labels are obtained based on the text metadata.

[0015] According to the above technical solution, the subtitle track is divided based on scene division, text metadata of the subtitle track is extracted according to the scenes, core content in the corresponding scene text metadata is selected through a fusion score function, and corresponding scene text labels are obtained.

[0016] As an optimization, S4, based on each scene label of different scenes, combines a weight component and a deviation value of a time point of each scene relative to a timeline, and comprehensively obtains a label of the video, and the specific method is as follows: According to a total time length of each scene relative to a total time length of the video, a weight coefficient is added to each label of each scene, a longest scene is taken as a main scene, a deviation value of a remaining scene relative to the main scene is taken as a correction of the weight coefficient, a same label in same type labels of each scene is adjusted and added to obtain a weight score of the scene label, and a plurality of scene labels with a highest ranking are output as video labels.

[0017] As an optimization, S4 further includes adding a narrative type label and a creation style label to the video through the visual emotion label or the auditory emotion classification label at different distribution positions and distribution frequencies of the video.

[0018] By the technical solution, the accuracy and interpretability of video tag generation can be improved through the spatiotemporal weight focusing and main scene anchoring mechanism, so that the video tag can reflect the core theme of the video, eliminate noise tags, and provide the video with information dimensions of narrative type tags and creative style tags.

[0019] In a second aspect, the application provides a system for implementing a video tag classification method based on metadata extraction and intelligent scene division, which specifically includes the following technical solutions: a video input module, a multi-modal analysis module, a metadata extraction module, a metadata structured storage module, and an interface output module; the video input module receives and manages input videos and converts them into intermediate format codes for storage; The multi-modal analysis module separates the video into image tracks, audio tracks, and subtitle tracks; The metadata extraction module includes a visual analysis submodule, an auditory analysis submodule, a text analysis submodule, and a fusion generation submodule; The visual analysis submodule extracts image metadata from the image tracks, performs scene division on the video based on the image metadata, and generates scene visual tags; The auditory analysis submodule extracts audio metadata from the audio tracks of each scene based on the scene division, and generates scene auditory tags; The text analysis submodule extracts text metadata from the subtitle tracks of each scene based on the scene division, and generates scene text tags; The fusion generation submodule combines the weight components and the deviation values of each scene tag in the video timeline based on the scene tags of different scenes to comprehensively obtain the video tags; The metadata structured storage module saves the video tag results output by the metadata extraction module as a structured format video information file, which is convenient for subsequent external system calling or external search engine indexing; The interface output module provides a standard API interface, pushes the structured format video information file to a third-party content management system, a recommendation system, or a search engine, or exports it as an external file for warehousing or archiving.

[0020] In a third aspect, the application provides a computer program product including a computer program or instructions, which enable the computer program or instructions to implement the steps of the above-described video tag classification method based on metadata extraction and intelligent scene division.

[0021] In summary, the application includes at least one of the following beneficial technical effects: 1.The application can identify the multi-dimensional content elements of a video by separating and extracting metadata of video tracks, audio tracks, and subtitle tracks, including not only static pictures but also sound, subtitles, semantic content, and obtaining a deeper understanding of video content information, identifying and defining video elements including emotions and themes, expanding the dimensions of video tags, and thus improving the accuracy of video search, association, and pushing, and improving user experience.

[0022] 2.The application compares the similarity of video intervals based on image metadata to achieve accurate video scene division, especially to label the scenes of long videos, solve the semantic generalization problem caused by labeling the whole video, refine video voice capture and tag definition, and retain long-tail tag information; users can jump to a specific scene through a specific tag to improve video viewing information acquisition efficiency.

[0023] 3.The application has high automation degree in processing input videos, greatly reduces the degree of human participation, and can significantly improve the efficiency of video information processing.

[0024] 4.The multi-dimensional metadata obtained by the application is finally output in a formatted video tag system, which is beneficial to retrieval and calling, has strong adaptability, and can be connected to third-party content management platforms or media systems. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 FIG. 1 is a flowchart of a video tag classification method based on metadata extraction and intelligent scene division according to an embodiment of the application; Figure 2 FIG. 2 is a flowchart of S3 in FIG. 1; Figure 3 FIG. 3 is a flowchart of S301 in FIG. 1; Figure 4 FIG. 4 is a schematic diagram of video interval similarity comparison using a sliding window in FIG. 1; Figure 5 FIG. 5 is a flowchart of S302 in FIG. 1; Figure 6 FIG. 6 is a flowchart of S303 in FIG. 1; Figure 7 FIG. 7 is an architecture diagram of a video tag classification system according to an embodiment of the application. DETAILED DESCRIPTION

[0026] The specific embodiments are merely explanatory of this application, and are not intended to limit this application. Those skilled in the art can make modifications to the embodiments according to the needs without creative contribution, and as long as the modifications are within the scope of this application, they are protected by the Patent Law.

[0027] To make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative contribution fall within the scope of protection of the present application. It should be noted that in the optional embodiments of the present application, the object information and other related data involved in the embodiments of the present application when applied to specific products or technologies need to obtain the permission or consent of the object, and the collection, use and processing of the related data need to comply with the relevant laws, regulations and standards of the country and region. That is, if the embodiments of the present application involve data related to the object, the data needs to be obtained with the authorization and consent of the object, the authorization and consent of the relevant department, and in compliance with the relevant laws, regulations and standards of the country and region. If the embodiments involve personal information, the consent of the individual needs to be obtained for the collection of all personal information, and the separate consent of the information subject needs to be obtained for the sensitive information, and the embodiments also need to be implemented with the authorization and consent of the object.

[0028] The embodiments of the present application will be described in further detail below with reference to the drawings of the specification.

[0029] Embodiment 1 Please refer to Figure 1 The video tag classification method based on metadata extraction and intelligent scene division in the embodiments of the present application includes the following steps: S1, the input video is acquired and transcoded, and is converted into an intermediate format code of H.264 / AVC format coding for storage. The length of the group of pictures (GOP) of H.264 / AVC format coding is set according to the length of the video.

[0030] The image group GOP is composed of I frames, P frames and B frames arranged in a certain order, the I frame is a key frame, the P frame and the B frame take the I frame as a reference frame, one image group GOP is composed of one I frame and several B / P frames, the length of the image group GOP is the number of frames between the I frame of one image group GOP and the I frame of the next image group GOP. The length of the image group GOP encoded by the H.264 / AVC format can be adjusted according to the parameters, so that when the input video is transcoded, the length of the image group GOP is adapted to the length of the video. A longer video uses a longer GOP, which can improve the compression rate of the video and reduce the code rate fluctuation caused by the insertion of the I frame, ensure the continuity of the video and reduce the cost of video storage and subsequent feature extraction calculation; a shorter video uses a shorter GOP, since a short video often contains faster editing and transition, scene switching is more frequent, a shorter GOP can ensure that high-frequency shots are accurately perceived, and the integrity of video information acquisition is improved.

[0031] S2, separating the video into image tracks, audio tracks and subtitle tracks.

[0032] After the video has been transcoded into the intermediate format, the image tracks, audio tracks and subtitle tracks of the video are extracted respectively, so that subsequent video metadata extraction in different dimensions can be carried out according to different tracks.

[0033] S3, sequentially extracting different modal metadata of the video from the image tracks, audio tracks and subtitle tracks respectively, and adding different dimensional labels to the video according to the metadata of the video.

[0034] It should be noted that the metadata of the video itself can be used as information representing the content of the video for storage, transmission and identification, but since different modal metadata have different dimensions, cross-modal connection problems may occur in actual application, therefore, by adding different dimensional labels to the video according to the metadata of the video, cross-modal semantic alignment of different modal metadata can be realized, the uniformity and easy understanding of video semantic expression are improved, and subsequent specific implementations such as video retrieval and recommendation using video labels are facilitated.

[0035] For details, please refer to Figure 2 , the video metadata extraction specifically includes: S301, extracting image metadata of the image track. The image metadata has two functions, one is to obtain the similarity of different video intervals based on the image metadata, and the other is to generate scene visual labels of each scene based on the image metadata.

[0036] More specifically, please refer to Figure 3 , the specific implementation method of the above steps includes: S3011, acquire a continuously arranged group of pictures (GOP) based on the intermediate format coding, and extract image metadata from the group of pictures (GOP).

[0037] Since the video has been transcoded into an intermediate coding format that adapts the video length, the length of the group of pictures (GOP) has been determined, and a single group of pictures (GOP) is the basic unit for image metadata extraction. The image metadata is a collection of multi-modal image metadata that extracts various types of features and semantic meanings in the group of pictures (GOP).

[0038] It should be noted that the group of pictures (GOP) is a frame sequence composed of an I frame and several B / P frames, which represents a video picture of a certain time length. Among them, the I frame is a full-frame compressed and encoded frame that encodes and transmits full-frame image information. When decoding, only the data of the I frame can reconstruct the complete image, which describes the details of the image background and moving objects, and does not need to refer to other pictures to generate; the P frame takes the I frame as the reference frame, finds the prediction value and motion vector of the P frame in the I frame, takes the prediction error and motion vector for encoding and transmission, and when decoding, the prediction value in the I frame must be summed with the prediction error to reconstruct the complete P frame image; the B frame takes the previous I or P frame and the subsequent P frame as the reference frame, which is a bidirectional prediction encoding frame.

[0039] When extracting the image metadata of the group of pictures (GOP), the picture acquisition information of the I frame is mainly extracted, and the motion vector and prediction residual of the P frame and B frame are combined to obtain multi-dimensional image metadata. The specific multi-dimensional image metadata extraction method is described in detail below.

[0040] S3012, scan the image track of the video using a sliding window with a preset width and interval to obtain a continuously arranged video interval, and any video interval is composed of several groups of pictures (GOP), and the image metadata of each group of pictures (GOP) forms the image metadata matrix of the video interval.

[0041] Since the content specifically described by the picture of the video in the time length direction and the time point of scene transition are unknown, the span of the group of pictures (GOP) may cover scene transitions or scene content information with large differences before and after. If a single key frame or a single group of pictures (GOP) is used as the basis for similarity comparison of scene transitions, large errors may occur.

[0042] Please refer to Figure 4In the present application, a technical solution of scanning the image track by using a sliding window with a preset width and interval is adopted, and one step of the sliding window is a video interval. The width of the sliding window means how many groups of pictures (GOPs) the sliding window is composed of, i.e., how many groups of pictures (GOPs) the video interval is composed of; and the interval of the sliding window means the number of groups of pictures (GOPs) that the sliding window steps each time. In the embodiments of the present application, the preset width of the sliding window is 3 groups of pictures (GOPs), and the interval is 3 groups of pictures (GOPs). When the interval of the sliding window is equal to the width of the sliding window, the video interval can cover all the image tracks in a head-to-tail manner. It should be noted that the interval of the sliding window does not necessarily equal the width of the sliding window, and can be adjusted as needed to make the video interval partially overlap.

[0043] Taking the video interval as the similarity comparison object of scene conversion can obtain all the video metadata of the groups of pictures (GOPs) within a certain time span, obtain a video metadata matrix, and take the video metadata matrix as the object of similarity comparison, which can ensure the semantic integrity and content continuity of the video interval, and ensure that the sliding window covers the time point boundary of scene switching, reduces the semantic expression difficulty of scene switching detection, and improves the scene switching detection accuracy.

[0044] S3013, the similarity of the image metadata matrix between adjacent video intervals is calculated, similarity judgment is performed, and if the similarity is lower than a threshold, it is judged that scene switching occurs, and the first group of pictures (GOP) in the subsequent video interval is defined as the time point of video scene switching. The timeline position of the I frame of the group of pictures (GOP) is marked to provide an anchor point for scene segmentation for subsequent data processing procedures.

[0045] S3014, for different scenes, scene visual labels are added to the scenes based on the image metadata.

[0046] More specifically, in the above steps, the groups of pictures (GOPs) are taken as objects to extract multi-dimensional image metadata, which specifically includes: The foreground and background of the group of pictures (GOP) are distinguished by using a dual-channel U-Net network.

[0047] Specifically, in the dual-channel U-Net network, the RGB image of the I frame of the group of pictures (GOP) is input into the spatial feature channel to extract the static appearance feature; and the inter-frame difference map generated by the P / B frame is input into the time sequence feature channel to quantize the motion vector. Through adaptive feature fusion of the attention mechanism, the background static area, the foreground target (person, object), and the transition boundary area of the foreground and the background of the group of pictures (GOP) are finally output.

[0048] Feature extraction is performed on the background to extract the edge pixel density, background texture and perspective relationship metadata of the background, and the background metadata is generated after feature fusion. The edge pixel density of the background reflects the structural complexity of the background, which can be obtained by generating a binary edge map through the Canny algorithm; the background texture is described by the spatial distribution rule of pixel grayscale, which is used to distinguish different materials or surfaces, and is obtained by a gray level co-occurrence matrix; the perspective relationship is used to quantify the depth information of the scene, and the three-dimensional space layout is inferred by analyzing the geometric structure. The above technical solutions are realized by applying existing algorithms, and will not be described here. Through the recognition of the background metadata, the environmental label of the scene can be obtained, such as indoor environment, urban street environment, forest environment, seaside beach environment, and cosmic environment.

[0049] The foreground is distinguished and recognized as a portrait and an object, a deep convolutional neural network is used to extract features of the portrait and the object respectively, and portrait metadata and object metadata are obtained. The portrait metadata includes face feature metadata and modeling feature metadata, which are used to anchor the portraits appearing in the video, and the portrait identity and the number of portraits can be used as a basis for judging scene switching. By recognizing the portrait metadata, the character label is obtained, including the gender, age, occupation, and clothing of the character; for a specific portrait, based on the celebrity database data, the specific celebrity identity can be anchored, and the corresponding celebrity label can be added to the video scene. The scene description label is obtained by recognizing the features of the object, such as recognizing the operating table and the shadowless lamp to obtain the operating room label, recognizing the hospital bed and the infusion stand to obtain the hospital room label, recognizing the bookshelf and the book to obtain the library label, and recognizing the desk and the blackboard to obtain the classroom label.

[0050] Feature extraction is performed on the picture color data to obtain picture color metadata, and visual emotion classification labels are obtained based on the picture color metadata. Specifically, a self-developed model, emotion-tone model STS (Style-Tone Scaler), is used to classify the emotional tendency of the video picture color statistical features.

[0051] Feature extraction is performed on the picture color data to obtain picture color metadata, and visual emotion classification labels are obtained based on the picture color metadata. Specifically, a self-developed model, emotion-tone model STS (Style-Tone Scaler), is used to classify the emotional tendency of the video picture color statistical features. c c c s s s c c c s wherein γ1, γ2, γ3, γ4 are adjustable weight parameters, and the calculation result obtains a unique solution E s ​​​​​​​​​​, visual emotion score E s mapping with visual emotion classification labels, the types of visual emotion classification labels are respectively Calm, Tense, Joyful, Melancholy and Intense.

[0052] After obtaining the multi-dimensional video metadata of the image group GOP, the multi-dimensional video metadata matrix of the video interval is obtained through the technical solution of the above sliding window scanning, and similarity comparison is performed. The specific method of similarity comparison of the video metadata matrix is the prior art, which can be realized by various different methods. In the embodiment of the present application, Manhattan distance is calculated to compare the similarity of multi-dimensional data matrix.

[0053] On the basis of realizing scene segmentation, environment tags are generated based on the background metadata, character tags are obtained based on the portrait metadata, scene description tags are obtained based on the object metadata, and visual emotion classification tags are obtained based on the picture color metadata, so as to add scene visual tags to each different scene of video segmentation.

[0054] On the basis of having realized scene segmentation, the extraction of metadata of audio tracks and subtitle tracks is continued. Specifically, it includes: S302, audio metadata of audio tracks of each scene is extracted according to the divided scenes, and scene auditory tags are generated. Please refer to Figure 5 , specifically including: S3021, according to the divided scenes, the audio tracks are segmented, and the audio tracks of each scene are extracted.

[0055] S3022, the background sound and the human voice of the audio track are separated. In the embodiment of the present application, NovaVSS tool can be selected to realize the separate extraction of the background sound and the human voice.

[0056] S3023, feature recognition is performed on the background sound to obtain environmental sound tags.

[0057] Specifically, in this embodiment, the environmental sound label is divided into different categories, including natural environment, urban environment, indoor environment and special environment. The natural environment includes forest, beach, mountain and rainforest, etc. The low-frequency natural sound is dominant, the energy changes gently, and the frequency spectrum is uniform. The urban environment includes street, subway station, shopping mall and construction area, etc. The medium and high frequency background human voice / mechanical sound is mixed, the energy changes frequently, such as car horn and crowd noise. The indoor environment includes office, restaurant, library and hospital, etc. The background human voice has high intelligibility, the background noise is stable, and the frequency spectrum is concentrated in 300-2000 Hz. The special environment includes concert, stadium and airport, etc. It has a high energy pulse, such as applause / engine sound, and a wide frequency band covering 20 Hz-20 kHz. It should be noted that the above background human voice is different from the human voice separated in S3022.

[0058] S3024, feature extraction is performed on the human voice to obtain an auditory emotion classification label. The specific method is: The feature extraction is performed on the human voice to obtain the pitch Pitch std and intensity Energy mean of the human voice metadata, and the auditory emotion score E h is calculated as shown in the following formula, E h = a Pitch std + b Energy mean wherein a and b are training parameters, which can be optimized through repeated test results and manual emotion judgment. The calculation result obtains a unique solution E h . The auditory emotion score E h forms a mapping with the auditory emotion classification label. The types of the auditory emotion classification label are calm, tense, joyful, melancholy and intense. The auditory emotion classification label is complementary and referenced to the visual emotion classification label, and the combination of the visual emotion label and the auditory emotion label forms a video emotion label.

[0059] S303, text metadata of the subtitle track of each scene is extracted according to the divided scenes to generate scene text labels. Please refer to Figure 6 , which specifically includes: S3031, text data of the subtitle track is extracted, and the text data is segmented according to the divided scenes.

[0060] S3032, word information extraction is performed on the text data to obtain independent word information w i . The semantic redundancy fusion algorithm is used to obtain the fusion score S(w i ) of each word information w i . The fusion score function is specifically: S(w i )=λ1·f(w i )+λ2·p(w i )+λ3·TF-IDF(w i ) wherein f(w i ) is a term frequency component score of the term information w i , obtained according to the frequency of occurrence of the term information w i ; p(w i ) is a position weight component score of the term information w i , assigned according to the position of occurrence of the term information w i in the text data; TF-IDF(w i ) is a specificity weight component score of the term information w i , obtained according to the co-occurrence frequency with other term information, to avoid the common words including “of” and “is” from being unable to be excluded by the term frequency component score f(w i ); λ1, λ2, λ3 are adjustable weight parameters.

[0061] S3033, the top-ranked term information w i according to the fusion score S(w i ) is screened as the text metadata of the subtitle track, and the scene text label summarizing the theme content of the video is obtained by recognizing the text metadata using the LLM large language model, to describe the content range and vertical field of the video.

[0062] In another embodiment, in the step S2 of separating the video into image tracks, audio tracks and subtitle tracks, the multi-source text metadata is obtained in a supplementary manner. Specifically, periodic OCR recognition is performed on the video frames to extract the independent subtitles and on-screen display (OSD) in specific regions of the picture as text metadata; the human voice extracted in S3023 is recognized using the Whisper model to obtain the text-to-speech processing result as text metadata.

[0063] S4, based on the scene labels of different scenes, combining the weight component of each scene in the video and the deviation value of the time point relative to the timeline, the video metadata extraction results of all modalities are integrated to automatically generate the labels of the video, including content summary, emotion label, theme, keyword, recommendation degree and other dimensions.

[0064] Specifically, according to the time of each scene in the total time of the video, a weight coefficient is added to each label of each scene, the longest scene is taken as the main scene, the deviation of the remaining scenes relative to the main scene is taken as the correction of the weight coefficient, the same labels in the same type of labels of each scene are weighted and added to obtain the weight score of the scene label, and the top ranked several scene labels are output as the video label.

[0065] If only the labels of each scene are simply superimposed, multiple secondary labels from different scenes will drown out the core label. By using the time weight and the deviation weight correction of the main scene, the real proportion of the label in the global video can be more accurately reflected, thereby improving the label accuracy and the user intention matching degree.

[0066] In another embodiment, the obtained label of the video further includes adding a narrative type label and a creation style label to the video by the visual emotional label and the auditory emotional classification label in different distribution positions, distribution frequencies and distribution manners of the video.

[0067] For example, if the Joyful emotional label of the video accounts for more than 60%, and the end scene is a Joyful emotional label, a comedy label is added to the video; if there is a Melancholy emotional label in the video, and the end scene is a Tense, Melancholy, or Intense emotional label, a tragedy label is added to the video; if the emotional label of the video scene is from Tense, Intense to Calm, Joyful, a healing direction label is added to the video; if the visual emotional label is Joyful and the auditory emotional label is Melancholy, a contrast montage label is added. The above embodiments are specific applications of adding labels to videos through emotional labels and scene distribution, and do not constitute a limitation on the technical solutions of the present application.

[0068] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0069] Embodiment 2: Please refer to Figure 7 A video label classification system of the embodiments of the present application is used to realize the video label classification method based on metadata extraction and intelligent scene division described above, which includes a video input module 1, a multi-modal analysis module 2, a metadata extraction module 3, a metadata structured storage module 4 and an interface output module 5.

[0070] The video input module 1 receives and converts the input video into H.264 / AVC format encoding for storage, and determines the length of the group of pictures (GOP) of the H.264 / AVC format encoding according to the video length. The video input module 1 supports various video media sources such as local upload, URL address, API interface, etc., and stores the downloaded and transcoded video in the local processing cache.

[0071] The multi-modal analysis module 2 separates the video into image tracks, audio tracks, and subtitle tracks.

[0072] The metadata extraction module 3 is the core module of the system, which uses various AI models to extract and analyze features of different modal information to generate video metadata. Specifically, it includes a visual analysis submodule 31, an auditory analysis submodule 32, a text analysis submodule 33, and a fusion generation submodule 34.

[0073] The visual analysis submodule 31 extracts image metadata from the image tracks, divides the video into scenes based on the image metadata, and generates scene visual labels.

[0074] The auditory analysis submodule 32 extracts audio metadata from the audio tracks of each scene based on the divided scenes, and generates scene auditory labels.

[0075] The text analysis submodule 33 extracts text metadata from the subtitle tracks of each scene based on the divided scenes, and generates scene text labels.

[0076] The fusion generation submodule 34 generates labels for the video based on the scene labels of different scenes, combines the weight components of each scene in the video and the deviation values of the appearance time points relative to the timeline, and comprehensively obtains the labels for the video.

[0077] In another embodiment, the metadata extraction module 3 further includes a structural analysis submodule 35 and an AI plugin submodule 36.

[0078] The structural analysis submodule 35 analyzes the structure of the video media file, extracts information such as encoding format, length, code rate, resolution, resolution, and sound channel, and provides information that can increase the dimensions of video metadata and video labels.

[0079] AI plugin sub-module 36: for the aforementioned extracted various types of intermediate video metadata, call the configured AI plugin for subdivision depth analysis, such as tracking updated vertical field data or current hot data. The plugin system supports the pre-training model access of the standard MCP standard interface specification, automatically delivers target metadata and pre-training configuration, and supports IPC gateway, shared memory, cache queue, database and other forms of plugin processing result return, which is convenient for extension and secondary development; when the vertical field or hot information lands, the self-owned Data-Agent (data agent) can be accessed through the plugin system, integrated into the local data set, and through more rich field data, the processing result is positively affected, and more representative analysis result is generated.

[0080] The metadata structured storage module 4 saves the video tag result output by the metadata extraction module as a structured format video information file, supports standard JSON / RDF, or other database structures, and is convenient for subsequent external system calling or external search engine indexing.

[0081] The interface output module 5 provides a standard API interface, supports pushing the structured format video information file to a third-party content management system, a recommendation system or a search engine, or exporting it as an external file for warehousing or archiving.

[0082] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the charging pile transaction processing system described above can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0083] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0084] Embodiment 3: The computer program product of the embodiment of the application includes a computer program or instructions, which enable the computer program or instructions to implement the steps of the above-mentioned video tag classification method based on metadata extraction and intelligent scene division.

[0085] In the above embodiments, all or part of the embodiments can be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments can be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. integrated with one or more available media. The available media can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk) and the like.

[0086] The above embodiments are only used to illustrate the technical solutions of the present application, but not limit the present application; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent replacements; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A video tag classification method based on metadata extraction and intelligent scene segmentation, characterized in that, Includes the following steps: S1: Acquire and transcode the input video, converting it into an intermediate format encoding for storage; S2 separates the video into image tracks, audio tracks, and subtitle tracks; S3 extracts the video's metadata, specifically including: S301, extract image metadata from the image track, obtain the similarity of different video intervals based on the image metadata, divide the video into scenes based on the similarity, and generate scene visual labels; S302, extract the audio metadata of the audio tracks of each scene according to the divided scenes, and generate scene auditory tags; S303, extract the text metadata of the subtitle track for each scene according to the divided scenes, and generate scene text tags; S4, based on the scene labels of different scenarios, combines the weight components of each scene in the video and the deviation of the time of appearance from the timeline to obtain the video's labels.

2. The video tag classification method based on metadata extraction and intelligent scene segmentation according to claim 1, characterized in that, The intermediate format encoding is H.264 / AVC format encoding, and the length of the Group of Pictures (GOP) encoded in H.264 / AVC format is set according to the length of the video.

3. The video tag classification method based on metadata extraction and intelligent scene segmentation according to claim 1, characterized in that, S301 involves extracting image metadata from the image track, obtaining similarity scores for different video intervals based on the image metadata, segmenting the video scenes based on the similarity scores, and generating scene visual labels, specifically including: S3011, Based on the intermediate format encoding, obtain a continuously arranged group of images (GOP), and extract image metadata using the group of images (GOP) as the object; S3012, a sliding window with a preset width and interval is used to scan the image track of the video to obtain continuously arranged video intervals. Any video interval is composed of several image groups (GOPs), and the image metadata of each image group (GOP) forms the image metadata matrix of the video interval. S3013, calculate the similarity of the image metadata matrix between adjacent video intervals, make a similarity judgment, and if the similarity is lower than the threshold, it is determined that a scene switch has occurred, and the first image group GOP in the subsequent video interval is defined as the time point of the video scene switch. S3014 adds scene visual tags to scenes based on image metadata, corresponding to different scenarios.

4. The video tag classification method based on metadata extraction and intelligent scene segmentation according to claim 3, characterized in that, The extraction of image metadata based on Group of Pictures (GOPs) specifically includes: A dual-channel U-Net network is used to distinguish the foreground and background of a group of images (GOPs). Feature extraction is performed on the background, extracting metadata such as edge pixel density, background texture, and perspective relationship of the background, and then performing feature fusion to generate background metadata; The foreground is distinguished and identified as people and objects. Features are extracted from people and objects respectively to obtain image metadata and object metadata. Feature extraction is performed on the color data of the image to obtain image color metadata; For different scenarios, scene visual tags are added to the scene based on image metadata, specifically including: An environment label is generated based on the background metadata, a person label is obtained based on the portrait metadata, a scene description label is obtained based on the object metadata, and a visual emotion classification label is obtained based on the image color metadata. The specific method for obtaining visual emotion classification labels based on image color metadata is as follows: Feature extraction is performed on the color data of the image to obtain the hue H. c Saturation S c Brightness L c and pixel density D s The visual emotion score E is calculated from the image color metadata. s As shown in the following formula, E s =γ1·H c +γ2·S c +γ3·L c +γ4·D s Where γ1, γ2, γ3, and γ4 are adjustable weight parameters, the calculation result yields a unique solution E. s Visual Emotion Score E s This is mapped to visual emotion classification labels, which are Calm, Tense, Joyful, Melancholy, and Intense.

5. The video tag classification method based on metadata extraction and intelligent scene segmentation according to claim 1, characterized in that, In step S302, audio metadata of the audio tracks is extracted according to the divided scenes to generate scene auditory tags, specifically including: S3021, Based on the divided scenes, the audio tracks are segmented and the audio tracks of each scene are extracted; S3022 separates background noise and vocals from the audio track; S3023, perform feature recognition on background noise to obtain environmental sound labels; S3024, extract features from human voice to obtain auditory emotion classification labels; The specific method for extracting features from human voices to obtain voice emotion classification labels is as follows: Feature extraction is performed on the human voice to obtain the pitch. std and intensity Energy mean Human voice metadata is used to calculate the auditory emotion score E. h As shown in the following formula, E h =a·Pitch std +β·Energy mean Where α and β are training parameters, the calculation result yields a unique solution E. h Auditory emotion score E h This is mapped to auditory emotion classification labels, which are Calm, Tense, Joyful, Melancholy, and Intense.

6. The video tag classification method based on metadata extraction and intelligent scene segmentation according to claim 1, characterized in that, S303 involves extracting the text metadata of the subtitle track for each scene according to the divided scenes, and generating scene text tags, specifically including: S3031, Extract the text data from the subtitle track and segment the text data according to the divided scenes; S3032, extract word information from the text data to obtain independent word information w. i A semantic redundancy fusion algorithm is used to obtain the information w of each word. i Fusion score S(w) i The fusion scoring function is as follows: S(w i )=λ1·f(w i )+λ2·p(w i )+λ3·TF-IDF(w i ) Where, f(w) i ) represents word information w i Word frequency component score, p(w i ) represents word information w i Location-weighted component scoring, TF-IDF(w i ) represents word information w i The specific weighted component scores are given, where λ1, λ2, and λ3 are adjustable weight parameters. S3033, Screening fusion score S(w) i The top-ranked words in the sorting information i As text metadata for the subtitle track, scene text tags are obtained based on the text metadata.

7. The video tag classification method based on metadata extraction and intelligent scene segmentation according to claim 1, characterized in that, S4, based on the scene labels of different scenes, and combining the weight components of each scene in the video and the deviation of its occurrence time from the timeline, comprehensively obtains the video labels. The specific method is as follows: Based on the time allotted to each scene in the total video duration, weight coefficients are added to each tag of each scene. The scene with the longest duration is designated as the main scene, and the deviations of the other scenes from the main scene are used to adjust the weight coefficients. The weighted sum of the same tags of the same type in each scene is used to obtain the weight score of the scene tag. The top-ranked scene tags are then output as video tags.

8. A video tag classification method based on metadata extraction and intelligent scene segmentation according to claim 4 or 5, characterized in that, S4, based on the scene tags of different scenes, combined with the weight components of each scene in the video and the deviation of the time of appearance from the timeline, comprehensively obtains the video tags. It also includes adding narrative type tags and creative style tags to the video by using the different distribution positions and frequencies of the visual emotion tags or the auditory emotion classification tags in the video.

9. A system for implementing a video tag classification method based on metadata extraction and intelligent scene segmentation, characterized in that, include: Video input module, multimodal parsing module, metadata extraction module, metadata structured storage module, and interface output module; The video input module receives and manages the input video, converts it into an intermediate format encoding, and stores it. The multimodal parsing module separates the video into image tracks, audio tracks, and subtitle tracks; The metadata extraction module includes a visual analysis submodule, an auditory analysis submodule, a text analysis submodule, and a fusion generation submodule; The visual analysis submodule extracts image metadata from the image track, divides the video into scenes based on the image metadata, and generates scene visual tags. The auditory analysis submodule extracts the audio metadata of the audio tracks of each scene according to the divided scenes, and generates scene auditory tags. The text analysis submodule extracts the text metadata of the subtitle track for each scene according to the divided scenes, and generates scene text tags; The fusion generation submodule, based on the scene tags of different scenes, combines the weight components of each scene in the video and the deviation of the time of appearance from the timeline to obtain the video tags. The metadata structured storage module saves the video tag results output by the metadata extraction module as a structured video information file, which is convenient for subsequent external system calls or for external search engine indexing. The interface output module provides a standard API interface to push structured video information files to third-party content management systems, recommendation systems, or search engines, or to export them as external files for storage or archiving.

10. A computer program product, characterized in that, The computer program product includes a computer program or instructions that enable the computer program or instructions to implement the steps of the video tag classification method based on metadata extraction and intelligent scene segmentation as described in any one of claims 1 to 7.

Images