A method and system for video segment tagging for scene cut detection

Abstract

The application discloses a scene change detection video segment label generation method and system, comprising: video segmenting, including using scene change detection to preliminarily identify scene change points in the video; dividing the video into segments with independent scenes, and extracting representative key frames from each scene segment; applying highlight detection algorithm to identify highlight areas in the video frame; finding the minimum point of the function of time and scene change score, thereby dividing the video into time-non-repeating segments, extracting static information and dynamic information in the video and labeling, thereby realizing the generation of video segment labels. The application also discloses a scene change detection video segment label generation system. The application can accurately divide the video into time-non-repeating segments, thereby greatly improving the management and retrieval efficiency of the video content, and enabling the user to more quickly find relevant video segments.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video understanding, and in particular to a scene transition detection-based video segment label generation method and system. BACKGROUND

[0002] In today's digital and information society, video content has become an important medium for information dissemination and entertainment. With the popularity of video platforms and the rapid growth of video data, we are facing a huge challenge: how to efficiently and accurately retrieve and process massive amounts of video content. Videos not only cover various types of entertainment programs, news reports, educational content, and user-generated content, but also involve constantly updated real-time data. These diverse video content increases the complexity of information retrieval and management, so traditional video retrieval methods, such as text label and keyword-based search, have become unsuitable for current needs.

[0003] Traditional text label and keyword search methods rely on text descriptions or labels provided by users for video content. However, this method has several obvious problems: first, text labels are often subjective and prone to omissions or mislabeling, which cannot fully and accurately describe video content; second, keyword search can only match existing labels and cannot handle unmarked content or details; finally, these methods are inefficient and have poor user experience when dealing with long videos or complex content. Therefore, finding a more intelligent and efficient video retrieval and processing method has become a pressing problem.

[0004] To address these challenges, the present patent proposes a video segment label generation method based on scene transition detection. This method uses advanced techniques to automatically segment and label video content, significantly improving the efficiency of video content retrieval and analysis.

[0005] One of the current existing technologies is a video segment label identification method and device (CN110263217A), which obtains a target video segment, extracts image and audio features of the segment, and uses a pre-trained multi-label classification model to analyze these features to obtain at least two-dimensional class labels for the target video segment. The invention also includes video splitting and highlight segment identification preprocessing steps to ensure that only highlight segments are processed to improve the relevance of the identification. The training process of the multi-label classification model includes feature extraction, label classification result analysis, and model parameter adjustment to ensure high accuracy of the model. The disadvantages of this invention are: this invention mainly extracts features and identifies labels for a single segment, and since this method only processes isolated segments, it may not fully utilize the contextual information in the entire video, which may result in incomplete or inaccurate understanding of the video content and label identification, especially in terms of video content continuity and integrity.

[0006] The second prior art is a video tag determination method and device (CN108694217B). The invention divides a video into multiple segments, extracts visual, audio and text features of each segment, generates candidate tags and their confidence based on the features, determines recommended tags in combination with information of adjacent segments, and finally generates a comprehensive tag of the video. The shortcomings of the invention are: the invention mainly focuses on feature extraction of video segments and generates tags based on these features, but ignores more detailed static information such as detailed visual information, audio information, subtitle information, etc. SUMMARY

[0007] The purpose of the present application is to overcome the shortcomings of the prior art and provide a scene change detection video segment tag generation method and system. The main problems solved by the present application are: 1) how to overcome the problems of inaccurate labeling, content omission and low retrieval efficiency in traditional video retrieval methods which usually rely on text tags and keyword search; 2) how to overcome the problem that the traditional method mainly relies on user-provided tags and keywords for understanding video content, which is prone to cause inaccurate information retrieval when facing complex or unlabeled video content, and the existing technology is difficult to accurately capture important information and details in the video, resulting in low relevance and accuracy of the retrieval results; 3) how to overcome the problem that traditional retrieval and analysis methods often cannot effectively process all data when facing long videos or complex video content, resulting in low processing efficiency; 4) how to overcome the problem that the traditional retrieval method cannot quickly and accurately provide the content required by the user, affecting the user's experience and satisfaction.

[0008] To solve the above problems, the present application provides a scene change detection video segment tag generation method, which comprises:

[0009] Video segment segmentation is performed, first using scene change detection to preliminarily identify scene change points in the video, second dividing the video into segments with independent scenes, using key frame detection to extract representative key frames from each scene segment, and applying highlight detection algorithm to identify highlight areas in the video frames, and finally finding the minimum point of the function of time and scene change score to divide the video into time-non-repeating segments;

[0010] According to the time-non-repeating segments, video static information containing visual information, audio information, subtitle information, emotional information and time information is extracted, and video dynamic information containing segment action and sound change is labeled, thereby realizing video segment tag generation.

[0011] Preferably, the scene change detection is used to preliminarily identify the scene change points in the video, and the video is divided into segments with independent scenes, specifically:

[0012] Feature extraction is used for the video, color histogram of each frame is calculated to describe the color distribution in the frame, texture analysis method is used to extract the texture feature of each frame, and the main object in the video is detected through the target detection algorithm;

[0013] The visual feature similarity between adjacent frames is calculated using the Euclidean distance, and the visual feature similarity between adjacent frames detected by the scene change detection is recorded to a one-dimensional array Ascore with a size of T-1, where T is the number of video frames; the jth element of this array represents the scene change score of the j+1th frame of the original video, and the lower the score, the higher the probability of the scene change point.

[0014] Preferably, the key frame detection is used to extract representative key frames from each scene segment, specifically:

[0015] Every K frames are selected as a frame, and the change of the picture content is used as the basis for selection to select the frame as a candidate key frame; K-means clustering is used to group similar frames, and the most representative frame is selected from each group, and the index I of these frames is recorded; Ascore[I] is set to ∞, which represents that this frame is a key frame and cannot be selected as a scene change point.

[0016] Preferably, the high light detection algorithm is applied to identify the high light area in the video frame, specifically:

[0017] Binary search is used for high light detection, and a high light detection model QVHighlights based on deep learning is used, in each high light detection process, a starting point Q and an ending point P are used as the model output results, representing the start and end frames of the high light area, respectively, and the two points Q and P are used to define the stable segment in the video, which is convenient for subsequent analysis;

[0018] The hyperparameters w and σ are set, where w represents the weighting value, which is used to adjust the influence of the high light detection segment on the visual feature similarity score Ascore, and σ represents the attenuation value after the binary search iteration, which controls the influence of the high light segment in Ascore gradually decreases with the increase of the iteration number, because the high light segment in the video is limited, and the more the iteration number, the lower the probability of the high light segment;

[0019] The high light detected segment is weighted and processed, and the corresponding value in Ascore is updated, specifically:

[0020] Ascore[i]=Ascore[i]+σ j w

[0021] where i∈[Q, P], represents each frame of the high light detection output, j is the iteration number, the weighting mechanism can ensure that the influence of the detected high light area on the visual feature similarity score is gradually attenuated in each iteration, allowing new high light information to be added in subsequent iterations;

[0022] The part [0, Q] before the high light area of the video segment and the part [P, T-1] after the high light area of the video segment are regarded as new video segments, and the two new segments are subjected to high light detection again to find other high light areas that may exist. The newly detected high light segments are weighted into Ascore again, and the process is repeated M times, where M is a pre-set fixed value.

[0023] Preferably, the minimum point of the function of time and scene change score is found, specifically:

[0024] Ascore is regarded as a function of time t and scene change score, and local minimum point detection is performed by comparing the scores of adjacent frames. If:

[0025] Ascore[t-1]>Ascore[t]<Ascore[t+1],

[0026] then t is a local minimum point, and:

[0027] t≠0andt≠T-1

[0028] where t-1 and t+1 represent the indices of the previous frame and the next frame of the current frame, respectively. For the first and last elements, they are directly excluded because they are the start frame and end frame of the video.

[0029] The indices t of the local minimum points that meet the conditions are stored in an array MIndices. If:

[0030] Ascore[0]<Ascore[1],

[0031] then 0 is a local minimum point, and if:

[0032] Ascore[T-2]>Ascore[T-1],

[0033] then T-1 is a local minimum point.

[0034] Determine the effective scene change point. The scene change point must be a local minimum point, and the local minimum point is not necessarily a scene change point. It needs to be checked by a threshold value. If the score is not low enough to be a scene change point, set the threshold value ∈, find the minimum point t of the video i such that:

[0035] Ascore[t i]<∈

[0036] t i ∈MIndices

[0037] The eligible t i are recorded as one of the scene transition points of the video, stored in an array MIndices, so as to realize the division of the video into time-unrepeatable segments.

[0038] Preferably, according to the time-unrepeatable segments, video static information including visual information, audio information, subtitle information, emotional information and time information is extracted and labeled, specifically:

[0039] The visual information of the video segment is extracted, including character labels, object labels, event labels and scene labels;

[0040] The character label is extracted, the main characters in the video are labeled by the target detection and recognition algorithm Yolov5, and the details of the characters are generated, including gender, clothing and skin color;

[0041] The object label is extracted, the main objects or objects in the video are identified and classified, and the object detection model Detectron2 is used to extract the object label;

[0042] The event label is extracted, the main events in the video are detected and marked, and the event recognition model SlowFast or event recognition model is used to mark the event;

[0043] The scene label is extracted, the scene is classified according to the visual content, and the scene classification model SceneNet is used to analyze and mark the scene type;

[0044] The audio information of the video segment is extracted, including the sound source label and the music name label;

[0045] The sound source label is extracted, the sound source in the audio is identified, and the sound classification model VGGish is used for sound source labeling;

[0046] The music name label is extracted, the background music is classified and labeled, and the music information retrieval technology is used to identify the music name;

[0047] The speech content in the video is extracted and labeled, the automatic speech recognition system Google Speech-to-Text API is used to transcribe the speech and generate labels, and different labels are generated through the transcription text of the dialogue and the speaker separation result. Specifically, when the text is uninterrupted, it indicates that one person speaks continuously without interruption, and it is labeled as monologue; when multiple people speak alternately in the text, it is labeled as dialogue; for background noise or non-verbal elements, it needs to be removed or labeled as background sound;

[0048] Extracting the subtitle information of the video segment, using natural language processing technology named entity recognition NER, part-of-speech tagging POS to extract nouns and entities, using keyword extraction algorithm TF-IDF and TextRank to analyze and label the subtitle content;

[0049] Extracting the emotional information of the video segment, performing sentiment analysis on the subtitle content and audio in the video to determine the emotional state, using the model BERT trained by the sentiment analysis training set for emotion classification;

[0050] Extracting the time information of the video segment, recording the start and end time points of each segment, i.e. the frame index in the scene transition point TransPoints, using the timestamp information of the video for labeling, based on the total length of the video and the relative position of the segment, labeling the segment as early, middle or late in the video.

[0051] Preferably, according to the time non-repeating segment, the video dynamic information containing the segment action and sound change is extracted and labeled, specifically:

[0052] Generate action tokens for each video segment, use the action recognition model SlowFast to extract and label the main action or behavior in the segment, convert the action features into tokens for recording to facilitate subsequent processing, analyze the visual changes in the video segment, use the optical flow method to capture dynamic changes and generate corresponding dynamic information tokens;

[0053] Generate sound change tokens for each audio segment, analyze and label the changes in the audio, use audio feature extraction technology to identify and record sound changes; analyze the changes in the speech content, use the speech analysis model to generate speech change tokens;

[0054] Integrate the dynamic information tokens in the video segment and the audio segment, integrate the action tokens of the video and the sound change tokens of the audio, generate complete dynamic information description, use dynamic information integration algorithm to integrate the dynamic features of all segments, thereby generating a global dynamic feature set containing the video dynamic information of the segment action and the sound change.

[0055] Correspondingly, the application also provides a scene transition detection video segment label generation system, comprising:

[0056] The video segmentation unit is used for video segment segmentation, which firstly uses scene change detection to preliminarily identify scene change points in the video, secondly divides the video into segments with independent scenes, uses key frame detection to extract representative key frames from each scene segment, and applies highlight detection algorithm to identify highlight areas in the video frame, and finally finds the minimum point of the function of time and scene change score, thereby dividing the video into time-non-repeating segments;

[0057] The information labeling unit is used for extracting and labeling video static information containing visual information, audio information, subtitle information, emotional information and time information, and video dynamic information containing segment action and sound change according to the time-non-repeating segments, thereby realizing the generation of video segment labels.

[0058] The present application has the following beneficial effects:

[0059] The present application can accurately divide the video into time-non-repeating segments through scene change detection, key frame extraction and highlight detection algorithm, thereby greatly improving the management and retrieval efficiency of video content. This fine segmentation can reduce redundant data and ensure that each segment represents an independent scene or event in the video.

[0060] The present application can provide comprehensive labels and descriptions for video segments by extracting visual, audio, subtitle and emotional information of the video and generating dynamic information tokens, which makes the analysis, retrieval and understanding of video content more comprehensive and supports more intelligent content management system.

[0061] The static information labels (such as characters, objects, events, etc.) and dynamic information labels (such as action tokens, sound change tokens) used in the present application enable video content to be retrieved based on fine-grained information, which improves the accuracy of the retrieval system and enables users to quickly find relevant video segments.

[0062] The present application integrates the dynamic characteristics of video and audio to generate complete dynamic information description, which helps to analyze the actions and sound changes in the video in depth, and this analysis capability is particularly important for video monitoring and behavior recognition application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0063] Figure 1 is a scene change detection video segment label generation method flow chart of an embodiment of the present application;

[0064] Figure 2 is a scene change detection video segment label generation system structure diagram of an embodiment of the present application. DETAILED DESCRIPTION

[0065] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative effort belong to the scope of protection of the present application.

[0066] Figure 1 is a flow chart of a video segment label generation method for scene change detection according to an embodiment of the present application, as shown in Figure 1 , the method comprises:

[0067] S1, video segment splitting is performed, first, scene change detection is used to preliminarily identify scene change points in the video, second, the video is divided into segments with independent scenes, key frame detection is used to extract representative key frames from each scene segment, and highlight detection algorithm is applied to identify highlight areas in the video frames, and finally, the minimum point of the function of time and scene change score is found, so as to divide the video into time-non-repeating segments;

[0068] S2, according to the time-non-repeating segments, video static information containing visual information, audio information, subtitle information, emotional information and time information, and video dynamic information containing segment action and sound change are extracted and labeled, so as to realize the generation of video segment labels.

[0069] Step S1 is specifically as follows:

[0070] S1-1: scene change detection is used to divide the video into segments with independent scenes. Feature extraction is used for the video, color histogram of each frame is calculated to describe color distribution in the frame, texture analysis method is used to extract texture features of each frame, and target detection algorithm is used to detect main objects in the video;

[0071] Euclidean distance is used to calculate visual feature similarity between adjacent frames, and visual feature similarity between adjacent frames obtained by using scene change detection is recorded to a one-dimensional array Ascore with a size of T-1, where T is the number of video frames; the jth element of the array represents the scene change score of the j+1th frame of the original video, and the lower the score, the higher the probability of appearing scene change points.

[0072] S1-2: video key frame extraction is performed. Every K frames select one frame, and use the change of picture content as the basis for selection to select the frame as a candidate key frame; similar frames are grouped by using K-means clustering, and the most representative frame is selected from each group, and the index I of the frame is recorded; Ascore[I] is set to ∞, which represents that the frame is a key frame and cannot be selected as a scene change point.

[0073] S1-3: Identify the highlight region in the video frame. The bisection method is used for highlight detection, and the highlight detection model QVHighlights based on deep learning is used. In each highlight detection process, a starting point Q and an ending point P are used as the model output results, representing the start and end frames of the highlight region, respectively. The two points Q and P are used to define a stable segment in the video, which facilitates subsequent analysis.

[0074] The hyperparameters w and σ are set, where w represents the weighting value, which is used to adjust the influence of the highlight detection segment on the visual feature similarity score Ascore, and σ represents the attenuation value after bisection iteration, which controls the gradual weakening of the influence of the highlight segment in Ascore with the increase of the iteration number. Since the highlight segments in the video are limited, the more iterations, the lower the probability of obtaining the highlight segment.

[0075] The detected highlight segments are weighted and the corresponding values in Ascore are updated, specifically:

[0076] Ascore[i] = Ascore[i] + σ j w

[0077] where i ∈ [Q, P] represents each frame output by the highlight detection, and j is the iteration number. This weighting mechanism ensures that the influence of the detected highlight region on the visual feature similarity score is gradually attenuated in each iteration, allowing new highlight information to be added in subsequent iterations.

[0078] The part [0, Q] before the highlight region of the video segment and the part [P, T-1] after the highlight region of the video segment are considered as new video segments, and the two new segments are subjected to highlight detection again to find other possible highlight regions. The newly detected highlight segments are again weighted into Ascore, and the process is repeated M times, where M is a pre-set fixed value.

[0079] S1-4: Find the minimum point of the function Ascore. Ascore is considered as a function of time t and scene transformation score, and local minimum point detection is performed by comparing the scores of adjacent frames. If:

[0080] Ascore[t-1] > Ascore[t] < Ascore[t+1],

[0081] then t is a local minimum point, and:

[0082] t ≠ 0 and t ≠ T-1

[0083] Wherein, t-1 and t+1 represent the indexes of the previous frame and the next frame of the current frame respectively, for the first and last elements, because it is the beginning frame and the end frame of the video, so it is directly excluded;

[0084] The index t of the qualified local minimum point is stored in an array MIndices, if:

[0085] Ascore[0]<Ascore[1],

[0086] 0 is a local minimum point, if:

[0087] Ascore[T-2]>Ascore[T-1],

[0088] T-1 is a local minimum point;

[0089] Determine the effective scene transition point, the scene transition point must be a local minimum point, the local minimum point is not necessarily a scene transition point, which needs to be tested by threshold, if the score is not low enough to make a scene transition point, set the threshold ∈, find the minimum point t of the video i So that:

[0090] Ascore[t i ]<∈

[0091] t i ∈MIndices

[0092] The qualified t i Is recorded as one of the scene transition points of the video, stored in an array MIndices, so as to realize the division of the video into time non-repeating segments.

[0093] Step S2, as follows:

[0094] S2-1: Extract video static information, including visual information, audio information, subtitle information, emotional information and time information, and label them.

[0095] S2-1-1: Extract the visual information of the video segment, including the labels of the characters, the labels of the objects, the labels of the events and the labels of the scenes;

[0096] Extract the character label, label the main characters in the video through the target detection and recognition algorithm Yolov5, and generate the details of the characters, including gender, clothing and skin color;

[0097] Extract the object label, identify and classify the main objects or objects in the video, and use the object detection model Detectron2 to extract the object label;

[0098] Extract event tags, detect and label major events in the video, label events through action recognition model SlowFast or event recognition model;

[0099] Extract scene tags, classify scenes based on visual content, use scene classification model SceneNet to analyze and label scene types;

[0100] S2-1-2: Extract audio information of video clips, including sound source tags and music name tags;

[0101] Extract sound source tags, identify sound sources in audio, use sound classification model VGGish for sound source annotation;

[0102] Extract music name tags, classify and label background music, use music information retrieval technology to identify music names;

[0103] Extract and label speech content in the video, use automatic speech recognition system Google Speech-to-Text API to transcribe speech and generate labels, generate different labels through the transcribed text of the dialogue combined with the speaker separation result, specifically, when the text is uninterrupted, it indicates that one person is speaking continuously, and no one else interrupts, it is labeled as monologue; When multiple people speak alternately in the text, it is labeled as dialogue; For background noise or non-verbal elements, they need to be removed or labeled as background sound;

[0104] S2-1-3: Extract subtitle information of video clips, use natural language processing technology named entity recognition NER, part-of-speech tagging POS to extract nouns and entities, use keyword extraction algorithm TF-IDF and TextRank to analyze and label the subtitle content;

[0105] S2-1-4: Extract emotional information of video clips, perform sentiment analysis on the subtitle content and audio in the video to determine the emotional state, use the model BERT trained on the sentiment analysis training set for sentiment classification;

[0106] S2-1-5: Extract time information of video clips, record the start and end time points of each clip, i.e. frame index in TransPoints, use video timestamp information for labeling, based on the total length of the video and the relative position of the clip, label the clip as early, middle or late in the video.

[0107] S2-2: Extract and label video dynamic information, including clip action and sound changes.

[0108] S2-2-1: generate action tokens for each video segment, use the action recognition model SlowFast to extract and label the main action or behavior in the segment, convert the action features into token records for subsequent processing, analyze the visual changes in the video segment, use the optical flow method to capture dynamic changes and generate corresponding dynamic information tokens;

[0109] S2-2-2: generate sound change tokens for each audio segment, analyze the changes in the audio and label them, use audio feature extraction techniques to identify and record sound changes; analyze the changes in the speech content, and use a speech analysis model to generate speech change tokens;

[0110] S2-2-3: integrate dynamic information tokens in video segments and audio segments, integrate action tokens in videos and sound change tokens in audios to generate complete dynamic information descriptions, use dynamic information integration algorithms to integrate dynamic features of all segments, thereby generating a global dynamic feature set containing video dynamic information of the segment actions and the sound changes.

[0111] Correspondingly, the application also provides a video segment label generation system for scene change detection, as shown in Figure 2 The system comprises:

[0112] a video segmentation unit 1 for video segment segmentation, which first uses scene change detection to preliminarily identify scene change points in the video, then divides the video into segments with independent scenes, uses key frame detection to extract representative key frames from each scene segment, and applies highlight detection algorithms to identify highlight areas in the video frames, and finally finds the minimum point of the function of time and scene change score, thereby dividing the video into time-non-repeating segments;

[0113] an information labeling unit 2 for extracting and labeling video static information containing visual information, audio information, subtitle information, emotional information, and time information, and video dynamic information containing segment actions and sound changes according to the time-non-repeating segments, thereby realizing the generation of video segment labels.

[0114] Therefore, the application can accurately divide the video into time non-repeated segments by scene change detection, key frame extraction and highlight detection algorithm, thereby greatly improving the management and retrieval efficiency of the video content, the fine division can reduce redundant data and ensure that each segment represents an independent scene or event in the video; the application can provide comprehensive labels and descriptions for the video segments by extracting the visual, audio, subtitle and emotional information of the video and generating dynamic information tokens, which makes the analysis, retrieval and understanding of the video content more comprehensive and supports a more intelligent content management system; the static information labels (such as characters, objects, events, etc.) and dynamic information labels (such as action tokens, sound change tokens) used by the application enable the video content to be retrieved based on fine-grained information, which improves the information richness of the retrieval system and enables users to quickly find related video segments; the application integrates the dynamic characteristics of the video and audio to generate complete dynamic information descriptions, which is helpful for in-depth analysis of the actions and sound changes in the video, and this analysis capability is particularly important for video monitoring, behavior recognition and other application scenarios.

[0115] The above describes in detail the scene change detection video segment label generation method and system provided by the embodiment of the application, and the principle and implementation manner of the application are described by applying specific examples; the above embodiment description is only used to help understand the method of the application and its core idea; meanwhile, for those skilled in the art, the specific implementation manner and application range can be changed according to the idea of the application; in summary, the content of the specification should not be understood as a limitation of the application.

Claims

1. A method for video segment tagging for scene cut detection, characterized in that, The method comprises: Video segment segmentation is performed, first using scene change detection to preliminarily identify scene change points in the video, second dividing the video into segments with independent scenes, using key frame detection to extract representative key frames from each scene segment, and applying a highlight detection algorithm to identify highlight regions in the video frames, and finally finding a minimum point of a function of time and scene change scores to divide the video into time-non-repeating segments; According to the time-non-repeating segments, video static information containing visual information, audio information, subtitle information, emotional information and time information, and video dynamic information containing segment action and sound change are extracted and labeled to realize generation of video segment labels; Specifically, the scene change detection is used to preliminarily identify scene change points in the video, and the video is divided into segments with independent scenes, specifically: Feature extraction is used for the video, color histograms of each frame are calculated to describe color distribution in the frame, texture features of each frame are extracted using a texture analysis method, and objects in the video are detected through an object detection algorithm; Euclidean distance is used to calculate visual feature similarity between adjacent frames, and visual feature similarity between adjacent frames obtained by using scene change detection is recorded to a one-dimensional array Ascore with a size of T-1, where T is the number of video frames; the jth element of the array represents the scene change score of the j+1th frame of the original video, and the lower the score, the higher the probability of a scene change point; Specifically, the highlight detection algorithm is applied to identify highlight regions in the video frames, specifically: Binary search is used for highlight detection, and a highlight detection model QVHighlights based on deep learning is used, in each highlight detection process, a starting point Q and an ending point P are used as model output results, representing the starting and ending frames of the highlight region, and the two points Q and P are used to define stable segments in the video, facilitating subsequent analysis; Hyperparameters w and σ are set, where w represents a weighting value for adjusting the influence of the highlight detection segment on the visual feature similarity score Ascore, and σ represents an attenuation value obtained by binary search iteration, and σ controls the gradual weakening of the influence of the highlight segment in Ascore with the increase of the number of iterations, because the number of highlight segments in the video is limited, and the more the number of iterations, the lower the probability of the highlight segment obtained; The highlight detected segment is subjected to weighted processing, and the corresponding value in Ascore is updated, specifically: Ascore[i] = Ascore[i] + σ j w Where i∈[Q,P] represents each frame output by the highlight detection, and j is the number of iterations, the weighted processing can ensure that the influence of the detected highlight region on the visual feature similarity score is gradually attenuated in each iteration, allowing new highlight information to be added in subsequent iterations; The part [0,Q] before the highlight region of the video segment and the part [P,T-1] after the highlight region of the video segment are regarded as new video segments, and the two new segments are subjected to highlight detection again to find other highlight regions that may exist, the newly detected highlight segments are weighted into Ascore again, and the process is repeated M times, where M is a fixed value set in advance.

2. The video segment tag generation method for scene change detection of claim 1, wherein, The representative key frames are extracted from each scene segment using key frame detection, specifically: Every K frames, use the change of picture content as the basis for selection, select a frame as a candidate key frame; use K-means clustering to group similar frames, and select the most representative frame from each group, record the index I of these frames; Set Ascore[I] to ∞, which represents that this frame is a key frame, because it is impossible to be a scene transition point, so it will not be selected.

3. The video segment tag generation method for scene change detection of claim 1, wherein, The minimum point of the function of time and scene transition score is found, specifically: Ascore is regarded as a function of time t and scene transition score, local minimum point detection is performed, and local minimum points are detected by comparing the scores of adjacent frames, if: Ascore[t-1]>Ascore[t]<Ascore[t+1], t is a local minimum point, and: t≠0andt≠T-1 Where t-1 and t+1 represent the index of the previous frame and the next frame of the current frame respectively, for the first and last elements, because they are the beginning frame and the end frame of the video, they are directly excluded; The index t of the local minimum point that meets the condition is stored in an array MIndices, if: Ascore[0]<Ascore[1], 0 is a local minimum point, if: Ascore[T-2]>Ascore[T-1], T-1 is a local minimum point; Determine the effective scene change point, the scene change point must be a local minimum point, the local minimum point is not necessarily a scene change point, and needs to be tested by a threshold value. If the score is not low enough to make a scene change point, set a threshold value ε, find the minimum point t of the video i So that: Ascore[t i ]<∈ The t i The scene change points of the video are stored in an array MIndices, thus enabling the video to be divided into temporally non-repeating segments.

4. The video segment tag generation method for scene change detection of claim 1, wherein, According to the time non-repeating segment, the video static information containing visual information, audio information, subtitle information, emotional information and time information is extracted and labeled, specifically: Extract the visual information of the video segment, including the person label, the object label, the event label and the scene label; Extract the person label, label the person in the video through the target detection and recognition algorithm Yolov5, and generate the details of the person, including gender, clothing and skin color; Extract the object label, identify and classify the objects or objects in the video, and use the object detection model Detectron2 to extract the object label; Extract the event label, detect and mark the events in the video, and mark the events through the action recognition model SlowFast or the event recognition model; Extract the scene label, classify the scene according to the visual content, and use the scene classification model SceneNet to analyze and mark the scene type; Extract the audio information of the video segment, including the sound source label and the music name label; Extract the sound source label, identify the sound source in the audio, and use the sound classification model VGGish to mark the sound source; Extract the music name label, classify and label the background music, and use music information retrieval technology to identify the music name; Extracting the speech content in the video and labeling, transcribing the speech using the automatic speech recognition system Google Speech-to-Text API and generating labels, generating different labels through the transcription text of the dialogue, combining the speaker separation results, specifically, when the text is uninterrupted, indicating a person's continuous speech, and no one else interrupts, it is labeled as monologue; when multiple people speak alternately in the text, it is labeled as dialogue; for background noise or non-verbal elements, it needs to be removed or marked as background sound; Extracting the subtitle information of the video segment, using natural language processing techniques such as Named Entity Recognition (NER) and Part-of-Speech Tagging (POS) to extract nouns and entities, using keyword extraction algorithms such as TF-IDF and TextRank to analyze and label the subtitle content; Extracting the emotional information of the video segment, performing sentiment analysis on the subtitle content and audio in the video to determine the emotional state, using a model trained on a sentiment analysis training set, BERT, for sentiment classification; Extracting the time information of the video segment, recording the start and end time points of each segment, i.e. the frame index of the scene transition point TransPoints, using the timestamp information of the video to mark, based on the total length of the video and the relative position of the segment, marking whether the segment is in the early, middle or late stage of the video.

5. The video segment tag generation method for scene change detection of claim 1, wherein, According to the time-unrepeated segment, extract and label the video dynamic information containing segment actions and sound changes, specifically: Generate action tokens for each video segment, use the action recognition model SlowFast to extract and label actions or behaviors in the segment, convert action features to token records; analyze the visual changes in the video segment, use the optical flow method to capture dynamic changes and generate corresponding dynamic information tokens; Generate sound change tokens for each audio segment, analyze and label changes in the audio, use audio feature extraction techniques to identify and record sound changes; analyze changes in speech content, use a speech analysis model to generate speech change tokens; Integrate dynamic information tokens in video segments and audio segments, integrate action tokens of the video and sound change tokens of the audio, generate complete dynamic information descriptions, use dynamic information integration algorithms to integrate dynamic features of all segments, thereby generating a global dynamic feature set containing video dynamic information of segment actions and sound changes.

6. A video segment tag generation system for scene cut detection, characterized by, The system includes: A video segmentation unit for video segment segmentation, first using scene transition detection to preliminarily identify scene transition points in the video, then dividing the video into segments with independent scenes, using key frame detection to extract representative key frames from each scene segment, and applying highlight detection algorithms to identify highlight areas in video frames, finally finding the minimum point of the function of time and scene transition score, thereby dividing the video into time-unrepeated segments; The information labeling unit is configured to extract and label video static information including visual information, audio information, subtitle information, emotional information and time information, and video dynamic information including segment action and sound change according to the time-unrepeated segment, so as to realize generation of a video segment label. Specifically, in the video segmentation unit, scene change detection is used to preliminarily identify scene change points in the video, and the video is divided into segments with independent scenes, specifically as follows: Feature extraction is used for the video, color histograms of each frame are calculated to describe color distribution in the frame, and texture analysis is used to extract texture features of each frame, and an object detection algorithm is used to detect objects in the video. Euclidean distance is used to calculate visual feature similarity between adjacent frames, and visual feature similarity between adjacent frames obtained by using scene change detection is recorded to a one-dimensional array Ascore with a size of T-1, where T is the number of video frames; the jth element of the array represents the scene change score of the j+1th frame of the original video, and the lower the score, the higher the probability of a scene change point. Specifically, in the video segmentation unit, a highlight detection algorithm is applied to identify highlight regions in the video frame, specifically as follows: Binary method is used for highlight detection, and a highlight detection model QVHighlights based on deep learning is used, in each highlight detection process, a starting point Q and an ending point P are used as model output results, representing the start and end frames of the highlight region, respectively, and the two points Q and P are used to define stable segments in the video, facilitating subsequent analysis; Hyperparameters w and sigma are set, where w represents a weighting value for adjusting the influence of the highlight detection segment on the visual feature similarity score Ascore, and sigma represents an attenuation value obtained by binary iteration, and sigma controls the gradual weakening of the influence of the highlight segment in Ascore with the increase of the number of iterations, because the number of highlight segments in the video is limited, and the more the number of iterations, the lower the probability of the highlight segment obtained; The highlight segments detected by the highlight detection are weighted and processed, and the corresponding values in Ascore are updated, specifically as follows: Ascore[i] = Ascore[i] + σ j w Where i [Q, P] represents each frame output by the highlight detection, and j is the number of iterations, the weighting processing can ensure that the influence of the detected highlight region on the visual feature similarity score is gradually attenuated in each iteration, and new highlight information is allowed to be added in subsequent iterations; The part [0, Q] before the highlight region of the video segment and the part [P, T-1] after the highlight region of the video segment are regarded as new video segments, and the two new segments are subjected to highlight detection again to find other highlight regions that may exist, the newly detected highlight segments are weighted into Ascore again, and the process is repeated M times, where M is a fixed value set in advance.

7. A video segment tag generation system for scene cut detection as claimed in claim 6, wherein, In the video segmentation unit, key frame detection is used to extract representative key frames from each scene segment, specifically as follows: Every K frames, select a frame, use the change of picture content as the basis for selection, select the frame as the candidate key frame; using K-means clustering to group similar frames, and select the most representative frame from each group, record the index I of these frames; Set Ascore[I] to ∞, to represent this frame as a key frame, because it is impossible to be a scene transition point, so it will not be selected.

8. The video segment tag generation system for scene cut detection of claim 6, wherein, In the video segmentation unit, find the minimum point of the function of time and scene transition score, specifically: Ascore is regarded as a function of time t and scene transition score, local minimum point detection is performed, and local minimum points are detected by comparing the scores of adjacent frames, if: Ascore[t-1]>Ascore[t]<Ascore[t+1], Then t is a local minimum point, and: t≠0andt≠T-1 Where t-1 and t+1 represent the index of the previous frame and the next frame of the current frame respectively, for the first and last elements, because they are the beginning frame and end frame of the video, they are directly excluded; The index t of the local minimum point that meets the condition is stored in an array MIndices, if: Ascore[0]<Ascore[1], Then 0 is a local minimum point, if: Ascore[T-2]>Ascore[T-1], Then T-1 is a local minimum point; Determine the effective scene change point, the scene change point must be a local minimum point, the local minimum point is not necessarily a scene change point, and needs to be tested by a threshold value. If the score is not low enough to make a scene change point, set a threshold value ε, find the minimum point t of the video i So that: Ascore[t i ]<∈ The t i The scene change points of the video are stored in an array MIndices, thus enabling the video to be divided into temporally non-repeating segments.

9. The video segment tag generation system for scene cut detection of claim 6, wherein, In the information labeling unit, according to the video segment, extract and label the video static information containing visual information, audio information, subtitle information, emotional information and time information, specifically: Extract the visual information of the video segment, including person label, object label, event label and scene label; Extract the person label, label the person in the video through the target detection and recognition algorithm Yolov5, and generate the details of the person, including gender, clothing and skin color; Extract the object label, identify and classify the objects or objects in the video, and use the object detection model Detectron2 to extract the object label; Extract the event label, detect and mark the events in the video, and mark the events through the action recognition model SlowFast or the event recognition model; Extract the scene label, classify the scene according to the visual content, and use the scene classification model SceneNet to analyze and mark the scene type; Extract the audio information of the video segment, including the sound source label and the music name label; Extract the sound source label, identify the sound source in the audio, and use the sound classification model VGGish to mark the sound source; Extract the music name label, classify and label the background music, and use music information retrieval technology to identify the music name; Extracting the speech content in the video and labeling, using the automatic speech recognition system Google Speech-to-Text API to transcribe the speech and generate labels, through the transcription text of the dialogue, combining the speaker separation results, different labels are generated, specifically, when the text is uninterrupted, indicating a person's continuous speech, and no one else interrupts, it is labeled as monologue; when the text appears to have multiple people speaking alternately, it is labeled as dialogue; for background noise or non-verbal elements, it needs to be removed or marked as background sound; Extracting the subtitle information of the video segment, using natural language processing technology named entity recognition NER and part-of-speech tagging POS to extract nouns and entities, using the keyword extraction algorithm TF-IDF and TextRank to analyze and label the subtitle content; Extracting the emotional information of the video segment, performing sentiment analysis on the subtitle content and audio in the video to determine the emotional state, using the model BERT trained on the sentiment analysis training set for sentiment classification; Extracting the time information of the video segment, recording the start and end time points of each segment, i.e. the frame index of the scene transition point TransPoints, using the timestamp information of the video to mark, based on the total length of the video and the relative position of the segment, marking whether the segment is in the early, middle or late stage of the video.

10. The video segment tag generation system for scene cut detection of claim 6, wherein, In the information labeling unit, according to the video segment, the video dynamic information containing the segment action and sound change is extracted and labeled, specifically: Generate action tokens for each video segment, use the action recognition model SlowFast to extract and label actions or behaviors in the segment, convert action features to token records; analyze the visual changes in the video segment, use the optical flow method to capture dynamic changes and generate corresponding dynamic information tokens; Generate sound change tokens for each audio segment, analyze and label the changes in the audio, use audio feature extraction technology to identify and record sound changes; analyze the changes in the speech content, use the speech analysis model to generate speech change tokens; Integrate the dynamic information tokens in the video segment and the audio segment, integrate the action tokens of the video and the sound change tokens of the audio, generate complete dynamic information description, use dynamic information integration algorithm to integrate the dynamic features of all segments, thereby generating a global dynamic feature set containing the video dynamic information of the segment action and the sound change.

Images