Video content intelligent management method and system based on multi-modal analysis

Abstract

The invention discloses a video content intelligent management method and system based on multi-modal analysis, and belongs to the field of video content management.The method comprises the steps that multi-modal features of video content are extracted; performing fine structure segmentation on the video content based on the multi-modal features, and segmenting the video into fine segments related to semantics; the fragments obtained after fine structure segmentation are subjected to tagging, and metadata is generated for each fragment; constructing a multi-modal semantic index of the video content based on the generated metadata; according to the method, a semantic retrieval function is provided by utilizing multi-modal semantic indexing, fine fragments required by a user are accurately recalled in combination with text label matching, measurement numerical value scoring, cross-modal vector retrieval and filtering based on a shot language and a shooting technology label, and the user queries and positions a specific time point or the fine fragments in a video through a natural language. According to the invention, the accuracy of video content analysis is improved, the user experience is optimized, and the operation complexity is reduced.

Description

Technical Field

[0001] The present invention relates to the field of video content management, and more specifically, to a video content intelligent management method and system based on multimodal analysis. Background Art

[0002] With the rapid growth of video content, traditional single-modality video management methods can no longer meet the efficient needs of content retrieval, editing, and analysis. Modern video management requires in-depth content understanding and labeling combined with multimodal data (such as vision, voice, text, etc.) to achieve accurate retrieval, intelligent editing, and personalized recommendations. However, most current systems have technical difficulties in multimodal data fusion analysis and visualization, making it difficult to fully tap the value of video content.

[0003] Furthermore, the prior art has the following technical problems:

[0004] 1) Low labeling efficiency: Traditional video content management methods rely on manual labeling or single-modal analysis, which is inefficient and error-prone, and cannot meet the management needs of massive videos.

[0005] 2) Insufficient retrieval precision: Analysis methods using a single modality (such as text or audio) are difficult to fully reflect the content of the video, resulting in insufficient relevance and accuracy of the retrieval results.

[0006] 3) Unable to accurately locate: It is difficult for existing systems to directly locate specific time points or segments in the video from retrieval results, which increases the complexity of user operations.

[0007] 4) Lack of refined structural segmentation: The video content cannot be segmented and structured in a refined manner, resulting in coarse granularity in semantic analysis and indexing, which affects usage efficiency.

[0008] 5) Lack of multimodal fusion and intelligent support: Unable to fully utilize the comprehensive information of visual, audio and text features, insufficient understanding of content, and low level of intelligence in the operation process. Summary of the invention

[0009] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a method and system for intelligent management of video content based on multimodal analysis. Through technical improvement and multimodal fusion, the accuracy of video content analysis is significantly improved, the user experience is optimized, the operation complexity is reduced, and efficient retrieval and management are achieved. It can be widely used in multiple fields such as media asset management, video production and editing, and realizes efficient management, accurate retrieval and positioning of large-scale video content.

[0010] The object of the present invention is achieved through the following solutions:

[0011] A video content intelligent management method based on multimodal analysis comprises the following steps:

[0012] Step S1, extracting multimodal features of video content;

[0013] Step S2, performing fine structural segmentation on the video content based on the multimodal features, dividing the video into semantically related fine segments;

[0014] Step S3, labeling the segments after the refined structure segmentation, and generating metadata for each segment;

[0015] Step S4, constructing a multimodal semantic index of the video content based on the generated metadata;

[0016] Step S5, using multimodal semantic indexing to provide semantic retrieval function, combining text label matching, metric numerical scoring, cross-modal vector retrieval, and filtering based on lens language and shooting technology labels, accurately recalls the fine segments required by users, and users locate specific time points or fine segments in the video through natural language queries.

[0017] Furthermore, in step S1, the multimodal features include visual features, audio features and text features; wherein, the extraction of visual features is based on the visual transformer model to extract deep features for each frame, and combines the scene detection algorithm to calibrate the scene switching points; the extraction of audio features is based on speech recognition and voiceprint analysis, and key acoustic features and classification are extracted through speech recognition, sentiment analysis and background sound features in the audio signal; the extraction of text features is based on natural language processing, and optical character recognition and natural language processing are performed on the subtitle information embedded in the video to extract semantic information and generate a text description.

[0018] Furthermore, in step S2, the video content is segmented into fine structures based on multimodal features, which specifically includes the following sub-steps:

[0019] Step S21, detecting scene changes based on visual features, and using scene switching points in the visual features to segment the video into preliminary segments;

[0020] Step S22, combining audio and text features, performing semantic similarity analysis on adjacent segments, and optimizing the segments based on audio features and semantic consistency;

[0021] Step S23, selecting a key frame with the best quality as a representative frame for each segment as a visual representative of the segment, and obtaining a refined video structure with semantic annotations and key frames.

[0022] Further, in step S21, the scene change detection based on the visual features and the scene switching points in the visual features are used to segment the video into preliminary segments, which specifically includes the following sub-steps:

[0023] Step 1, shot segmentation: segmenting the video shots based on time or changing visual features; the changing visual features include any one of color histogram comparison, optical flow analysis, and image feature point matching;

[0024] Step 2, fine segmentation: Based on the shot segmentation, each shot is finely segmented into fine segments Snippets; specifically, the following sub-steps are included:

[0025] Step 2.1, extract feature vectors and quality scores frame by frame: In each shot, extract the feature vector F(f i ), where f i Represents the i-th frame in the shot:

[0026] F(f i )=CNN(f i );

[0027] Next, a quality score is assigned to each frame, which takes into account factors such as picture clarity, character clarity, noise and blur. The quality score (Q(f i ))The following comprehensive scoring formula is used:

[0028] Q(f i )=α·C(f i )+β·P(f i )-γ·N(f i )-δ·M(f i );

[0029] Among them, C(f i ) is the clarity score, P(f i ) is the character clarity score, N(f i ) is the noise score, M(f i ) is the blur score, (α, β, γ, δ) are adjustment coefficients, indicating the importance of each factor in the picture quality score;

[0030] Step 2.2, filter out frames with poor quality by rating: For all frames in each shot, rate their image quality, with the rating criteria covering clarity, character clarity, noise, and blur;

[0031] Sharpness: Sharpness is measured by calculating the Laplace transform of the image or using edge detection, as follows:

[0032]

[0033] in, is the gradient calculation of the i-th frame, N is the total number of pixels, C(fi ) is the clarity score;

[0034] Character clarity: For images containing people, evaluate the character clarity; use face detection and key point detection algorithms to score the local clarity of faces and characters;

[0035] Noise and blur: Noise and blur are scored. The noise is calculated using the PSNR or SSIM indicator to measure the blur score, and the blur score is evaluated by calculating the FWHM of the image:

[0036]

[0037] Among them, M(f i ) represents the fuzziness score, is the absolute value of the image gradient;

[0038] If the quality score of a frame is lower than a certain predetermined threshold, the frame is discarded to prevent the low-quality picture from affecting subsequent segmentation;

[0039] By calculating the quality score Q(f i ), filter out frames with poor quality; low-quality frames are removed before aggregation to ensure that the final generated fine fragments have higher visual quality; the screening criteria set a minimum score threshold Q min , when Q(f i )<Q min , the frame is discarded;

[0040] Step 2.3, calculate the similarity of the picture feature vectors for aggregation: After filtering out the frames that meet the quality standards, calculate the feature similarity of these frames, and aggregate the adjacent frames into a fine snippet based on the similarity; the similarity calculation uses the similarity measure between feature vectors, and for frame f i and f j , and its similarity is calculated by the following formula:

[0041]

[0042] Among them, S(f i , f j ) is frame f i and f j The cosine similarity of i ) and F(f j ) are frame f i and f j The extracted feature vector, |F(f i )| and |F(f j )| are the L2 norms of the feature vectors, and the calculated similarity value S(f i, f j ) is used to measure the similarity between frames. If the similarity is high, it means that these frames have similar visual content and are merged into the same fine segment Snippet;

[0043] Step 2.4, Snippet generation: Based on the results of similarity calculation, the frames with high similarity are aggregated into a Snippet. When the similarity S(f i , f j ) exceeds a certain set threshold S thresh When , these frames will be aggregated into a fine segment, the formula is as follows:

[0044] Snippet k =f i |S(f i , f j )≥S thresh , f j ∈Shot i ;

[0045] Among them, Snippet k is the kth fine segment, f i and f j is frame i and frame j, S(f i , f j ) is the frame f i and frame f j The similarity between thresh is the threshold of similarity, which controls the aggregation standard of similar frames. When the similarity is lower than the set threshold, it is considered that there are significant differences between the frames and switches to a new fine segment;

[0046] Step 2.5, generate the final fine segment: After quality screening and similarity analysis, each shot is divided into several fine segments Snippets. Each fine segment contains images with high consistency in vision and quality, which is suitable for subsequent labeling, analysis and retrieval; the final result of fine segment segmentation is expressed as:

[0047] Shot i =Snippet 1 , Snippet 2 , …, Snippet k ;

[0048] Among them, Shot i is the i-th shot, Snippet 1 , Snippet 2 , ..., Snippet k are the k fine segments cut from the shot;

[0049] Step 3, filtering: After obtaining several cluster combinations in each shot, the cluster results are filtered according to the duration threshold, and only the cluster results that meet the length requirement are retained as the final generated fine segments.

[0050] Furthermore, in step S3, in the process of labeling the video clips after the refined structure segmentation, the label of each video clip is confirmed according to multiple dimensions, specifically including: confirming the visual element label, the lens language and shooting technology label, the sound label and the cross-modal vector.

[0051] Furthermore, in step S3, labeling the video segments after the refined structure segmentation specifically includes the following sub-steps:

[0052] Step 1: People and object recognition: Use convolutional neural networks and target detection algorithms to identify people and objects in the video in real time;

[0053] Step 2, behavior and scene analysis: Use behavior recognition algorithms to identify dynamic behaviors in the video, and analyze the background through scene recognition networks;

[0054] Step 3: Sound classification and speech recognition: Classify and transcribe the audio in the video through audio signal processing and acoustic models;

[0055] Step 4: Cross-modal vector generation: A cross-modal encoding model is used to map visual or audio information and text information into the same space, generating a cross-modal vector to support joint retrieval.

[0056] Step 5: Color and light measurement value detection: Use a computer vision model to perform detection and obtain a value between 0 and 100.

[0057] Furthermore, in step S5, the multimodal semantic index is used to provide a semantic search function, and a fine segment search combining text tags with vectors and numerical values ​​is realized. The user locates a specific time point or segment in the video through a natural language query, which specifically includes the following sub-steps:

[0058] Step 1, filtering based on lens language and shooting technology tags: In the initial stage of the retrieval process, fine clips are filtered by the lens language and shooting technology tags provided by the user. For each fine clip, the lens language and shooting technology tags are pre-labeled and stored to form an inverted index and define a set of filtering conditions; the filtering conditions are matched in the tag set of the fine clips, and only the clips that meet all the conditions are retained; if there is no completely matching clip, the match is downgraded, and the important parameters set by the user are given priority;

[0059] Step 2, similarity scoring based on metric values: for the filtered segments, similarity scoring is performed based on metric values, wherein the metric values ​​include hue, color temperature and light direction;

[0060] Hue: The color histogram (H 片段 ) and user expected color distribution (H 目标 ) similarity calculation score, using the cosine similarity formula:

[0061]

[0062] Color temperature: According to the target color temperature range specified by the user, the color temperature value (T 片段 ) and the target value (T 目标 ) using the Gaussian function:

[0063]

[0064] Light direction: Using the angle difference calculation method, the fragment light direction angle (θ 片段 ) and the user's desired direction angle (θ 目标 ) as the score:

[0065] Score 光线方向 =cos(θ 片段 -θ 目标 ).

[0066] A video content intelligent management system based on multimodal analysis, comprising:

[0067] Multimodal feature extraction module, used to extract visual features, audio features and text features of the video;

[0068] The structural segmentation module is used to perform fine segmentation of the video based on the extracted multimodal features;

[0069] A metadata generation module, used to generate metadata for each segmented fragment, wherein the metadata includes key frames, text descriptions, sentiment analysis, and semantic tags;

[0070] The retrieval module is used to achieve multimodal matching based on semantic similarity, combining visual, audio and text information to provide relevance ranking of retrieval results. Users input query requests to retrieve and locate target fine segments in the video;

[0071] The visual interaction module is used to intuitively display the retrieval results and video content through a timeline or graphical interface.

[0072] Furthermore, it also includes an automatic annotation module for batch processing video library content and generating high-quality semantic metadata.

[0073] Furthermore, in the visualization interaction module, multimodal information of the video clip is displayed based on the timeline, including key frames, text labels and emotion change curves.

[0074] The beneficial effects of the present invention include:

[0075] (1) The present invention improves the accuracy of video content analysis: by integrating visual, audio and text multimodal information, it achieves deep semantic understanding of video content and avoids errors caused by single modality analysis. At the same time, it introduces refined video structure segmentation technology to perform segment-level semantic annotation on the content, ensuring accurate stratification and high-precision expression of the content.

[0076] (2) The present invention optimizes the user interaction experience: through semantic search and intuitive visual interfaces such as timeline and relationship map, users can quickly locate the required content and simplify the operation process. The search results can be directly associated with the specific time point or segment of the video to achieve accurate playback, greatly improving the fluency and convenience of the user experience.

[0077] (3) The present invention reduces the workload of video editing: it automatically generates semantic tags, key frames and segment descriptions, reducing the need for manual annotation and analysis. Refined segmentation and multimodal analysis provide structured references for video editing, significantly shortening content screening and editing time.

[0078] (4) The present invention achieves efficient content retrieval and positioning: it supports semantic retrieval based on natural language, combined with refined segmentation, to quickly find the target content and accurately locate the corresponding segment. It optimizes the content indexing mechanism, improves the retrieval speed of large-scale video libraries, and achieves millisecond-level response and efficient positioning.

[0079] (5) The present invention improves the intelligence level of the system: by automatically processing video content through deep learning models, the intelligence and autonomy of the system are enhanced. The analysis results are automatically generated by combining multimodal data to achieve end-to-end intelligent management and optimization.

[0080] (6) The present invention supports large-scale video library management: the efficient storage and indexing mechanism combined with multimodal semantic analysis provides high-performance support for the management of large-scale video content. At the same time, the refined segment and metadata generation capabilities enhance the system's performance in scenarios such as video classification, retrieval, and recommendation. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0082] Figure 1 The fine segment structure and upper-lower level relationship diagram obtained after multimodal analysis of the video in the embodiment of the present invention;

[0083] Figure 2 A flowchart of video fine segment generation and indexing in an embodiment of the present invention;

[0084] Figure 3 is a processing flow chart of video fine segment retrieval in an embodiment of the present invention;

[0085] Figure 4 The figure is a flow chart of the method steps of the present invention. DETAILED DESCRIPTION

[0086] All features disclosed in all embodiments in this specification, or steps in all methods or processes implicitly disclosed, except for mutually exclusive features and / or steps, can be combined and / or expanded or replaced in any manner.

[0087] The specific implementation process of the present invention is as follows:

[0088] In a preferred embodiment of the present invention, a video content intelligent management method based on multimodal analysis is provided, which uses multimodal information such as vision, audio and text to analyze and finely segment the video, thereby achieving structured management and accurate retrieval. Figure 4 As shown, the specific steps include:

[0089] Step S1, extracting multimodal features of video content, including visual features, audio features, and text features;

[0090] Step S2, performing refined structural segmentation on the video content based on the multimodal features, and dividing the video into semantically related segments;

[0091] Step S3, generating metadata for each segment, wherein the metadata includes key frames, text descriptions, sentiment analysis, and semantic tags;

[0092] Step S4, constructing a multimodal semantic index of the video content based on the generated metadata;

[0093] Step S5 provides a semantic search function, and the user can quickly locate a specific time point or segment in the video through natural language query.

[0094] In a further embodiment, in step S2, if Figure 1 and Figure 2 As shown in the figure, the extraction of visual features is based on the visual transformer (ViT) model to extract deep features for each frame, and combines it with the scene detection algorithm to calibrate the scene switching points; the extraction of audio features is based on speech recognition and voiceprint analysis, and key acoustic features and classification are extracted through speech recognition, sentiment analysis and background sound features in the audio signal; the extraction of text features is based on natural language processing technology, and optical character recognition (OCR) and natural language processing (NLP) are performed on the subtitle information embedded in the video to extract semantic information and generate text descriptions.

[0095] In a further embodiment, the refined structure segmentation includes the following steps:

[0096] Step S21, detecting scene changes based on visual features, specifically using scene switching points in the visual features to segment the video into preliminary segments;

[0097] Step S22, combining audio and text features, performing semantic similarity analysis on adjacent segments, and further optimizing segment division based on audio features and semantic consistency;

[0098] In step S23, a key frame with the best quality is selected as a representative frame for each segment as a visual representation of the segment. The final result is a refined video structure with semantic annotations and key frames, which is convenient for subsequent retrieval and analysis.

[0099] It should be further explained that in the process of video content segmentation, the video processing flow is divided into two main steps: first, the video is segmented into basic shots, and then each shot is finely segmented. In the fine segmentation stage, the system not only screens according to the picture quality, but also combines the segments by the picture similarity, thereby generating several fine segments with high quality and consistency. The following is a detailed description of the segmentation process in the present invention:

[0100] Step 1. Shot cutting

[0101] Shot segmentation of a video is a basic segmentation step based on time or scene changes. A shot is a video segment that starts from a set of continuous images and ends with the next scene switch or shot transition (such as cut, dissolve, etc.). This process relies on computer vision technology, which detects the shot boundaries in the video and divides the video into several shots based on the changing visual features (such as scene changes, color differences, motion direction, etc.).

[0102] The core algorithm of this step can be implemented by selecting the following methods according to the characteristics of the specific video type:

[0103] Color Histogram Comparison: Shot boundaries are identified by computing the color histogram differences between adjacent frames.

[0104] Optical flow analysis: Motion detection (such as optical flow) is used to find the motion differences between adjacent frames and then detect shot boundaries.

[0105] Image feature point matching: Use feature matching methods (such as SIFT) to analyze the similarity between frames. When the similarity is lower than the set threshold, it is considered that a shot switch has occurred.

[0106] Step 2. Refine segmentation

[0107] Based on the shot segmentation, each shot is then segmented into fine-grained snippets. The goal of fine-grained snippet segmentation is to further subdivide each shot into smaller, more representative snippets for the convenience of subsequent tagging, retrieval, and analysis. This step mainly includes the following important processing steps:

[0108] First, feature vectors of each shot are extracted and quality scored frame by frame. Then, frames with poor quality are filtered out based on the score. Finally, frames are aggregated by calculating feature similarity between frames, and aggregation results with too short duration are discarded to generate high-quality Snippets.

[0109] Step 2.1 Extract feature vectors and quality scores frame by frame

[0110] In each shot, the system first extracts feature vectors for each frame. Feature vector extraction can use pre-trained deep neural networks (such as VGG, etc.) to extract high-level semantic features of the image. These feature vectors represent rich visual information in the image, including color, shape, texture, etc.

[0111] Feature extraction: The feature vector F(f i ), where f i Represents the i-th frame in the shot.

[0112] F(f i )=CNN(f i );

[0113] Next, each frame is assigned a quality score that takes into account factors such as image clarity, character clarity, noise, and blur. i ) Using the aforementioned comprehensive scoring formula:

[0114] Q(f i )=α·C(f i )+β·P(f i )-γ·N(fi )-δ·M(f i );

[0115] Among them, C(f i ) is the clarity score, P(f i ) is the character clarity score, N(f i ) is the noise score, M(f i ) is the blur score, (α, β, γ, δ) are adjustment coefficients, indicating the importance of each factor in the picture quality score.

[0116] Step 2.2 Filter poor quality frames by rating

[0117] First, for all frames in each shot, the system will score their image quality, and the scoring criteria cover several aspects:

[0118] Clarity: The clarity of the image is evaluated by calculating the image's contrast, sharpness and other indicators to ensure that the details in the video are clearly displayed. Specifically, the clarity can be measured by calculating the Laplace transform of the image or using edge detection technology. The formula is as follows:

[0119]

[0120] in, is the gradient calculation of the i-th frame, N is the total number of pixels, and C(fi) is the clarity score.

[0121] Character clarity: For images containing people, the system will specifically evaluate the clarity of the people. The local clarity of faces and people is scored through face detection and key point detection algorithms (such as MTCNN, etc.). The specific method can measure the recognizability of people based on the contrast and edge sharpness of the image.

[0122] Noise and blur: Noise and blur in the picture will affect the visual quality, so noise and blur scores are required. The noise calculation can be measured using indicators such as PSNR or SSIM. The blur score can be evaluated by calculating the FWHM of the image:

[0123]

[0124] Among them, M(f i ) represents the fuzziness score, is the absolute value of the image gradient.

[0125] If the quality score of a frame is lower than a predetermined threshold, the system will discard the frame to prevent the low-quality picture from affecting subsequent segmentation.

[0126] By calculating the quality score Q(f i), which can filter out frames with poor quality. Low-quality frames are removed before aggregation to ensure that the final fine-grained fragments have higher visual quality. The screening criterion is usually to set a minimum score threshold Q min , when Q(f i )<Q min When the score is lower than 0.6, the frame will be discarded. For example, frames with a score lower than 0.6 can be regarded as low-quality frames and discarded.

[0127] Step 2.3 Calculate the similarity of the picture feature vectors for aggregation

[0128] After selecting the frames that meet the quality standards, the feature similarity of these frames is calculated, and the adjacent frames are aggregated into a fine snippet based on the similarity. The similarity is calculated using the similarity metric between feature vectors, such as cosine similarity. i and f j , its similarity can be calculated by the following formula:

[0129]

[0130] Where: S(f i , f j ) is frame f i and f j The cosine similarity of i ) and F(f j ) are frame f i and f j The extracted feature vector, |F(f i )| and |F(f j )| are the L2 norms of the eigenvectors.

[0131] The calculated similarity value S(f i , f j ) is used to measure the similarity between frames. If the similarity is high, it means that these frames have similar visual content and are suitable for merging into the same fine segment (Snippet).

[0132] Step 2.4Snippet generation

[0133] Based on the similarity calculation results, the system aggregates frames with high similarity into a Snippet. Specifically, when the similarity S(f i , f j ) exceeds a certain set threshold (S thresh ), these frames will be aggregated into a fine segment with the following formula:

[0134] Snippet k =f i|S(f i , f j )≥S thresh , f j ∈Shot i ;

[0135] Among them: Snippet k is the kth fine segment, f i and f j is frame i and frame j, S(f i ,f j ) is the frame f i and frame f j The similarity between thresh is the similarity threshold, which controls the aggregation criteria of similar frames.

[0136] When the similarity falls below a set threshold, the system considers that there are significant differences between the frames and therefore switches to a new fine segment.

[0137] Step 2.5 Generate the final fine fragment

[0138] After the above quality screening and similarity analysis, the system divides each shot into several fine segments (Snippets). Each fine segment contains images that are highly consistent in both visual and quality, suitable for subsequent labeling, analysis and retrieval. The final result of fine segment segmentation can be expressed as:

[0139] Shot i =Snippet 1 ,Snippet 2 ,…,Snippet k ;

[0140] Among them, Shot i is the i-th lens, (Snippet 1 ,Snippet 2 ,…,Snippet k ) are the k fine segments cut from the shot.

[0141] After the above steps, several clustering combinations can be obtained in each shot. To ensure that the final generated results are usable, the clustering results need to be filtered according to the time threshold, and only clustering results of sufficient length are retained as the final generated fine segments. These fine segments not only retain the core information of the video content, but also remove low-quality images, ensuring that both visual quality and information consistency meet subsequent analysis and retrieval requirements.

[0142] Through shot segmentation and refined segmentation, the system can generate high-quality and consistent refined segments (Snippets), which can not only retain the structure of the video content, but also meet the needs of subsequent video content analysis, labeling and retrieval. The picture quality scoring and similarity analysis in the refined segmentation process effectively remove low-quality pictures, ensuring that each refined segment has high visual quality and information consistency, thereby improving the accuracy and efficiency of subsequent processing.

[0143] In a further embodiment, in step S3, during the process of labeling the fine video segments, in the labeling phase, the system will automatically generate or manually confirm the label (and value, vector) of each video segment according to multiple dimensions, specifically including the following categories:

[0144] 1. Visual element labels:

[0145] People: Identify people in the video using face recognition technology.

[0146] Contains people: Mark whether the video clip contains people, suitable for multi-character scenes.

[0147] Character characteristics: including gender, age group, facial expression, clothing style and other characteristics.

[0148] Behavior: Identify the specific actions of people in the video.

[0149] Objects: Mark key objects that appear in the video, such as "car", "mobile phone", etc.

[0150] Subject Movement Direction: Mark the movement direction of a person or object.

[0151] Background: Describe the background content appearing in the video.

[0152] Landmark: A specific geographical location or building.

[0153] Time: Time information involved in the video, such as "night", "winter", etc.

[0154] Location: Identify the geographic location where the video was shot.

[0155] Shooting Subject: A general label for the subject of the video.

[0156] Overall description: A comprehensive label description of the video content, summarizing the content characteristics and background of the video.

[0157] 2. Lens language and shooting technique tags:

[0158] Shot type: Mark the shot type according to the content of the picture, such as "close-up" or "long shot".

[0159] Shooting techniques: record the shooting method or technology, such as "push", "pull", "shake", etc.

[0160] Lens movement direction: Mark the direction of lens movement, such as "translation", "rotation", etc.

[0161] Shot Angle: Label the angle from which the shot was taken, such as "High Angle Shot" or "Low Angle Shot."

[0162] Shooting method: describes the shooting method, such as "stabilizer shooting" and "aerial photography".

[0163] Composition method: Mark according to the composition rules in the picture, such as "rule of thirds", "symmetry", etc.

[0164] 3. Sound Tags:

[0165] Sound classification: Classify the sounds in the video, such as "music", "dialogue", etc.

[0166] Ambient sound: describes the ambient sound in the background of the video, such as "wind" and "water flow".

[0167] Speech to text: The text label generated after the speech content in the video is transcribed.

[0168] 4. Color and light related values:

[0169] Hue: The main color style in the video.

[0170] Brightness: A label for the light intensity of the video.

[0171] Color temperature: Describes whether the video image is cool or warm.

[0172] Light direction: Mark the direction and nature of the light source, such as "backlight", "side light", etc.

[0173] 5. Cross-modal vectors:

[0174] Cross-modal vectors: Generate cross-modal vectors for the visual and audio contents in videos. These vectors will help with text-visual or text-audio joint retrieval.

[0175] The specific technical implementation process of the labeling conceived by the present invention is as follows: the label generation is carried out by combining deep learning with image / audio processing technology. In each frame or each clip of the video, automatic label generation is performed through multimodal information fusion. The specific steps are as follows:

[0176] Step 1: Person and object recognition: Use convolutional neural networks (CNN) and target detection algorithms (such as YOLO and Faster R-CNN) to identify people and objects in the video in real time.

[0177] Step 2, behavior and scene analysis: Use behavior recognition algorithms (such as LSTM-based timing analysis networks) to identify dynamic behaviors in the video, and analyze the background through scene recognition networks (such as ResNet or EfficientNet).

[0178] Step 3: Sound classification and speech recognition: Classify and transcribe the audio in the video through audio signal processing technology and acoustic models (such as VGGish and Whisper).

[0179] Step 4: Cross-modal vector generation: Use a cross-modal encoding model (such as CLIP) to map visual or audio information and text information into the same space and generate a cross-modal vector to support joint retrieval.

[0180] Step 5: Color and light measurement value detection: Use a computer vision model (such as OpenCV) to perform detection and obtain a value between 0 and 100.

[0181] The present invention introduces a rich label system in video content management, including labels of multiple dimensions such as characters, objects, scenes, behaviors, shooting details, colors and light, and supports more accurate content retrieval and management through vector labels generated by cross-modal analysis technology. This labeling method will greatly improve the retrieval efficiency of video content, and is particularly suitable for the management and intelligent operation of large-scale video libraries.

[0182] In a further implementation, in step S5, fine segment retrieval combining text tags with vectors and numerical values ​​is implemented. Specifically, content retrieval is a core component of the video content intelligent management system. In order to support efficient and accurate retrieval, the retrieval module accurately recalls the fine segments (Snippet) required by users by combining text tag matching, metric numerical scoring, cross-modal vector retrieval, and filtering based on lens language and shooting technology tags. The following are the technical principles and implementation details of the retrieval method:

[0183] Step 1. Filter based on lens language and shooting technique tags

[0184] At the initial stage of the search process, the system filters the fine clips through the lens language and shooting technology tags provided by the user, thereby narrowing the search scope and improving the relevance of the results. User-specified tags include lens language and shooting technology. Among them, lens language mainly refers to the scene (such as long shot, medium shot, close-up), lens movement direction (such as push, pull, shake, shift). Shooting technology mainly refers to shooting method (such as drone shooting, handheld shooting), composition method (such as rule of thirds composition, symmetrical composition), vertical and horizontal shooting angles.

[0185] The filtering logic of the present invention is as follows: for each fine segment, the system pre-labels and stores the lens language and shooting technology labels to form an inverted index. When the user searches, a set of filtering conditions is defined. For example:

[0186] =Scenery: medium shot, shooting method: handheld, composition method: rule of thirds;

[0187] The system's tag collection in fine fragments Match the filter conditions in the filter to keep only the fragments that meet all the conditions:

[0188]

[0189] If there is no exact matching fragment, downgrade matching is supported, giving priority to important parameters set by the user.

[0190] Step 2. Proximity scoring based on metric values

[0191] For the clips that pass the filter, the similarity score is calculated based on the metric values ​​(such as hue, color temperature, light direction, etc.):

[0192] Hue: The color histogram (H 片段 ) and user expected color distribution (H 目标 ) similarity calculation score, using the cosine similarity formula:

[0193]

[0194] Color temperature: According to the target color temperature range specified by the user, the color temperature value (T 片段 ) and the target value (T 目标 ) using the Gaussian function:

[0195]

[0196] Light direction: Using the angle difference calculation method, the fragment light direction angle (θ 片段 ) and the user's desired direction angle (θ 目标 ) as the score:

[0197] Score 光线方向 =cos(θ 片段 -θ 目标 ).

[0198] In other preferred embodiments of the present invention, based on the above-mentioned video content intelligent management method, a video content retrieval method based on multimodal semantic indexing is provided, such as Figure 3As shown, this embodiment describes how to use multimodal semantic indexing to achieve efficient video content retrieval and positioning:

[0199] Step 1, multimodal semantic index construction: for the refined structure generated in Example 1, the system further establishes a semantic index for each segment;

[0200] Index content: including visual features (keyframes), audio features (emotional categories and audio spectra), and text features (descriptions and tags) of the clips;

[0201] Index format: It uses efficient vectorized representation to convert multimodal features into a unified embedding space, which facilitates similarity matching during retrieval.

[0202] Hierarchical management: Organize the fragments in the index according to the timeline order and semantic categories to achieve hierarchical management.

[0203] Step 2, search and locate: Users can initiate searches in the following ways:

[0204] Natural language query: Enter text containing semantic information, such as "sports award ceremony", and the system parses the query semantics and matches the corresponding fragments in the index;

[0205] Multimodal query: supports queries based on image or audio features. Users upload screenshots or audio clips of the target segment, and the system finds matching content through multimodal similarity calculation.

[0206] After the system returns the retrieval results, the user can directly click to jump to the specific time point of the target segment, or browse the multimodal information of the segment (such as key frames, text descriptions, and audio emotion tags).

[0207] In a preferred embodiment of the present invention, a video content intelligent management system based on multimodal analysis is provided, comprising:

[0208] Multimodal feature extraction module, used to extract visual features, audio features and text features of the video;

[0209] The structural segmentation module is used to perform fine segmentation of the video based on the extracted multimodal features;

[0210] The metadata generation module is used to generate semantic labels and descriptions for each segmented fragment;

[0211] A retrieval module, used to retrieve and locate a target segment in a video based on a query request input by a user;

[0212] The visual interaction module is used to intuitively display the retrieval results and video content through a timeline or graphical interface.

[0213] In a further embodiment, the retrieval module supports multimodal matching based on semantic similarity, combining visual, audio and text information to provide relevance ranking of retrieval results.

[0214] In a further embodiment, the visualization interaction module displays multimodal information of the video clip based on the timeline, including key frames, text labels and emotion change curves.

[0215] In a further implementation, an automatic annotation module is also included for batch processing the video library content to generate high-quality semantic metadata.

[0216] The units involved in the embodiments of the present invention may be implemented by software or hardware, and the units described may also be arranged in a processor. The names of these units do not, in some cases, limit the units themselves.

[0217] According to one aspect of an embodiment of the present invention, a computer program product or a computer program is provided, the computer program product or the computer program includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in the above various optional implementations.

[0218] As another aspect, an embodiment of the present invention further provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.

Claims

1. A video content intelligent management method based on multimodal analysis, characterized in that: The following steps are involved: Step S1, extracting multimodal features of video content; Step S2, performing fine structural segmentation on the video content based on the multimodal features, dividing the video into semantically related fine segments; Step S3, labeling the segments after the refined structure segmentation, and generating metadata for each segment; Step S4, constructing a multimodal semantic index of the video content based on the generated metadata; Step S5, using multimodal semantic indexing to provide semantic retrieval function, combining text label matching, metric numerical scoring, cross-modal vector retrieval, and filtering based on lens language and shooting technology labels, accurately recalls the fine segments required by users, and users locate specific time points or fine segments in the video through natural language queries.

2. The video content intelligent management method based on multimodal analysis according to claim 1 is characterized in that: In step S1, the multimodal features include visual features, audio features and text features; wherein, the extraction of visual features is based on the visual transformer model to extract deep features for each frame, and the scene switching points are calibrated in combination with the scene detection algorithm; the extraction of audio features is based on speech recognition and voiceprint analysis, and key acoustic features and classification are extracted through speech recognition, sentiment analysis and background sound features in the audio signal; the extraction of text features is based on natural language processing, and optical character recognition and natural language processing are performed on the subtitle information embedded in the video to extract semantic information and generate a text description.

3. The video content intelligent management method based on multimodal analysis according to claim 1 is characterized in that: In step S2, the video content is segmented into fine structures based on multimodal features, which specifically includes the following sub-steps: Step S21, detecting scene changes based on visual features, and using scene switching points in the visual features to segment the video into preliminary segments; Step S22, combining audio and text features, performing semantic similarity analysis on adjacent segments, and optimizing the segments based on audio features and semantic consistency; Step S23, selecting a key frame with the best quality as a representative frame for each segment as a visual representative of the segment, and obtaining a refined video structure with semantic annotations and key frames.

4. The video content intelligent management method based on multimodal analysis according to claim 3 is characterized in that: In step S21, the scene change detection based on visual features and the scene switching points in the visual features are used to segment the video into preliminary segments, which specifically includes the following sub-steps: Step 1, shot segmentation: segmenting the video shots based on time or changing visual features; the changing visual features include any one of color histogram comparison, optical flow analysis, and image feature point matching; Step 2, fine segmentation: Based on the shot segmentation, each shot is finely segmented into fine segments Snippets; specifically, the following sub-steps are included: Step 2.1, extract feature vectors and quality scores frame by frame: In each shot, extract the feature vector F(fi) of each frame through the pre-trained convolutional neural network CNN, where fi represents the i-th frame in the shot: F(f i )=CNN(f i ); Next, a quality score is assigned to each frame, which takes into account factors such as picture clarity, character clarity, noise and blur. The quality score (Q(f i ))The following comprehensive scoring formula is used: Q(f i )=α·C(fi)+β·P(f i )-γ·N(f i )-δ·M(f i ); Among them, C(f i ) is the clarity score, P(f i ) is the character clarity score, N(f i ) is the noise score, M(f i ) is the blur score, (α, β, γ, δ) are adjustment coefficients, indicating the importance of each factor in the picture quality score; Step 2.2, filter out frames with poor quality by rating: For all frames in each shot, rate their image quality, with the rating criteria covering clarity, character clarity, noise, and blur; Sharpness: Sharpness is measured by calculating the Laplace transform of the image or using edge detection, as follows: in, is the gradient calculation of the i-th frame, N is the total number of pixels, and C(fi) is the clarity score; Character clarity: For images containing people, evaluate the character clarity; use face detection and key point detection algorithms to score the local clarity of faces and characters; Noise and blur: Noise and blur are scored. The noise is calculated using the PSNR or SSIM indicator to measure the blur score, and the blur score is evaluated by calculating the FWHM of the image: Among them, M(f i ) represents the fuzziness score, is the absolute value of the image gradient; If the quality score of a frame is lower than a certain predetermined threshold, the frame is discarded to prevent the low-quality picture from affecting subsequent segmentation; By calculating the quality score Q(fi) of each frame, the frames with poor quality are screened out; low-quality frames are removed before aggregation to ensure that the final fine fragments have higher visual quality; the screening criteria set a minimum score threshold Q min , when Q(f i )<Q min , the frame is discarded; Step 2.3, calculate the similarity of the picture feature vectors for aggregation: After filtering out the frames that meet the quality standards, calculate the feature similarity of these frames, and aggregate the adjacent frames into a fine snippet based on the similarity; the similarity calculation uses the similarity measure between feature vectors, and for frame f i and f j , and its similarity is calculated by the following formula: Among them, S(f i , f j ) is frame f i and f j The cosine similarity of i ) and F(f j ) are frame f i and f j The extracted feature vector, |F(f i )| and |F(f j )| are the L2 norms of the feature vectors, and the calculated similarity value S(f i , f j ) is used to measure the similarity between frames. If the similarity is high, it means that these frames have similar visual content and are merged into the same fine segment Snippet; Step 2.4, Snippet generation: Based on the results of similarity calculation, the frames with high similarity are aggregated into a Snippet. When the similarity S(f i , f j ) exceeds a certain set threshold S thresh When , these frames will be aggregated into a fine segment with the following formula: Snippet k =f i |S(f i ,f j )≥S thresh ,f j ∈Shot i ; Among them, Snippet k is the kth fine segment, f i and f j is frame i and frame j, S(f i , f j ) is the frame f i and frame f j The similarity between thresh is the threshold of similarity, which controls the aggregation standard of similar frames. When the similarity is lower than the set threshold, it is considered that there are significant differences between the frames and switches to a new fine segment; Step 2.5, generate the final fine segment: After quality screening and similarity analysis, each shot is divided into several fine segments Snippets. Each fine segment contains images with high consistency in vision and quality, which is suitable for subsequent labeling, analysis and retrieval; the final result of fine segment segmentation is expressed as: Shot i =Snippet1,Snippet2,…,Snippet k ; Among them, Shot i is the i-th shot, (Snippet1, Snippet2, ..., Snippet k ) is the (k) fine segments cut from the shot; Step 3, filtering: After obtaining several cluster combinations in each shot, the cluster results are filtered according to the duration threshold, and only the cluster results that meet the length requirement are retained as the final generated fine segments.

5. The video content intelligent management method based on multimodal analysis according to claim 1 is characterized in that: In step S3, in the process of labeling the video clips after the refined structure segmentation, the label of each video clip is confirmed according to multiple dimensions, specifically including: confirming the visual element label, the lens language and shooting technology label, the sound label and the cross-modal vector.

6. The video content intelligent management method based on multimodal analysis according to claim 1 is characterized in that: In step S3, labeling the video segments after the refined structure segmentation specifically includes the following sub-steps: Step 1: People and object recognition: Use convolutional neural networks and target detection algorithms to identify people and objects in the video in real time; Step 2, behavior and scene analysis: Use behavior recognition algorithms to identify dynamic behaviors in the video, and analyze the background through scene recognition networks; Step 3: Sound classification and speech recognition: Classify and transcribe the audio in the video through audio signal processing and acoustic models; Step 4: Cross-modal vector generation: A cross-modal encoding model is used to map visual or audio information and text information into the same space, generating a cross-modal vector to support joint retrieval. Step 5: Color and light measurement value detection: Use a computer vision model to perform detection and obtain a value between 0 and 100.

7. The video content intelligent management method based on multimodal analysis according to claim 1 is characterized in that: In step S5, the multimodal semantic index is used to provide a semantic search function, and a fine segment search combining text tags with vectors and numerical values ​​is realized. The user locates a specific time point or segment in the video through a natural language query, which specifically includes the following sub-steps: Step 1: Filtering based on lens language and shooting technology tags: In the initial stage of the retrieval process, filter the fine clips based on the lens language and shooting technology tags provided by the user. For each fine clip, pre-label and store the lens language and shooting technology tags to form an inverted index and define a set of filtering conditions. Match the filtering conditions in the tag set of the fine clips and only retain the clips that meet all the conditions. If there is no exact matching fragment, the match is downgraded, giving priority to the important parameters set by the user; Step 2, similarity scoring based on metric values: for the filtered segments, similarity scoring is performed based on metric values, wherein the metric values ​​include hue, color temperature and light direction; Hue: The color histogram (H 片段 ) and user expected color distribution (H 目标 ) similarity calculation score, using the cosine similarity formula: Color temperature: According to the target color temperature range specified by the user, the color temperature value (T 片段 ) and the target value (T 目标 ) using the Gaussian function: Light direction: Using the angle difference calculation method, the fragment light direction angle (θ 片段 ) and the user's desired direction angle (θ 目标 ) as the score: Score 光线方向 =cos(θ 片段 -θ 目标 )。 8. A video content intelligent management system based on multimodal analysis, characterized in that: include: Multimodal feature extraction module, used to extract visual features, audio features and text features of the video; The structural segmentation module is used to perform fine segmentation of the video based on the extracted multimodal features; A metadata generation module, used to generate metadata for each segmented fragment, wherein the metadata includes key frames, text descriptions, sentiment analysis, and semantic tags; The retrieval module is used to achieve multimodal matching based on semantic similarity, combining visual, audio and text information to provide relevance ranking of retrieval results. Users input query requests to retrieve and locate target fine segments in the video; The visual interaction module is used to intuitively display the retrieval results and video content through a timeline or graphical interface.

9. The video content intelligent management system based on multimodal analysis according to claim 8, characterized in that: It also includes an automatic annotation module for batch processing of video library content and generating high-quality semantic metadata.

10. The video content intelligent management system based on multimodal analysis according to claim 8, characterized in that: In the visualization interaction module, multimodal information of the video clip is specifically displayed based on the timeline, including key frames, text labels and emotion change curves.

Images