Video clip label generation method and system for scene change detection

Abstract

The invention discloses a video clip label generation method and system for scene change detection, and the method comprises the steps: carrying out the segmentation of a video clip, and carrying out the preliminary recognition of a scene change point in a video through scene change detection; dividing the video into segments with independent scenes, and extracting representative key frames from each scene segment; using a highlight detection algorithm to identify a highlight area in the video frame; and searching a minimum point of a function of the time and the scene change score, so that the video is divided into segments with non-repeated time, and static information and dynamic information in the segmented video segments are extracted and marked, so that the generation of video segment labels is realized. The invention further discloses a video clip label generation system for scene change detection. According to the method, the video can be accurately segmented into fragments which are not repeated in time, so that the management and retrieval efficiency of the video content is greatly improved, and a user can more quickly find related video fragments.

Description

Technical Field

[0001] The present invention relates to the field of video understanding, and in particular to a method and system for generating video segment labels for scene change detection. Background Art

[0002] In today's digital and information-based society, video content has become an important medium for information dissemination and entertainment. With the popularity of video platforms and the rapid growth of video data, we are facing a huge challenge: how to efficiently and accurately retrieve and process massive amounts of video content. Video not only covers various types of entertainment programs, news reports, educational content, and user-generated content, but also involves constantly updated real-time data. These diverse video contents increase the complexity of information retrieval and management. Therefore, traditional video retrieval methods, such as text tag and keyword-based search, no longer meet current needs.

[0003] Traditional text tag and keyword search methods rely on text descriptions or tags provided by users for video content. However, this method has several obvious problems: first, text tags are often subjective, prone to omissions or mislabeling, and cannot fully and accurately describe video content; second, keyword search can only be matched based on existing tags and cannot handle unlabeled content or details; finally, these methods have low retrieval efficiency and poor user experience when facing long videos or complex content. Therefore, finding a more intelligent and efficient video retrieval and processing method has become an urgent problem to be solved.

[0004] In order to meet these challenges, this patent proposes a method for generating video segment labels based on scene change detection. This method uses advanced technical means to automatically segment and label video content, thereby significantly improving the efficiency of video content retrieval and analysis.

[0005] One of the current prior arts is a method and device for identifying video clip labels (CN110263217A). The invention obtains a target video clip, extracts the image and audio features of the clip, and uses a pre-trained multi-label classification model to analyze these features to obtain the category labels of the target video clip in at least two dimensions. The invention also includes preprocessing steps such as video segmentation and highlight clip identification to ensure that only highlight clips are processed to improve the relevance of identification. The training process of the multi-label classification model includes feature extraction, label classification result analysis and model parameter adjustment to ensure the high accuracy of the model. The disadvantage of the invention is that the invention mainly performs feature extraction and label identification on a single clip. Since the method only processes isolated clips, it may not be able to fully utilize the contextual information in the entire video, which may result in the understanding of the video content and the identification of labels being not comprehensive or accurate, especially in terms of the continuity and integrity of the video content.

[0006] The second existing technology is a method and device for determining video labels (CN108694217B). This invention divides the video into multiple segments, extracts the visual, audio and text features of each segment, generates candidate labels and their confidences based on the features, and combines the information of adjacent segments to determine the recommended labels, and finally generates a comprehensive label for the video. The disadvantage of this invention is that it mainly focuses on the feature extraction of video segments and generates labels based on these features, but ignores more detailed static information, such as detailed visual information, audio information, subtitle information, etc. Summary of the invention

[0007] The purpose of the present invention is to overcome the shortcomings of existing methods and propose a method and system for generating video clip labels for scene change detection. The main problems solved by the present invention are: 1) How to overcome the problem that traditional video retrieval methods usually rely on text tags and keyword searches, and have problems such as inaccurate labeling, missing content and low retrieval efficiency; 2) How to overcome the problem that traditional methods mainly rely on user-provided tags and keywords for understanding video content, which easily causes inaccurate information retrieval when facing complex or unlabeled video content, and the existing technology is difficult to accurately capture important information and details in the video, resulting in low relevance and accuracy of retrieval results; 3) How to overcome the problem that traditional retrieval and analysis methods are often unable to effectively process all data when facing long videos or complex video content, resulting in low processing efficiency; 4) How to overcome the problem that traditional retrieval methods cannot quickly and accurately provide users with the content they need, affecting user experience and satisfaction.

[0008] In order to solve the above problem, the present invention proposes a method for generating video segment labels for scene change detection, the method comprising:

[0009] To segment video clips, first use scene change detection to preliminarily identify scene change points in the video, then divide the video into clips with independent scenes, use key frame detection to extract representative key frames from each scene clip, and apply highlight detection algorithm to identify highlight areas in video frames, and finally find the minimum point of the function of time and scene change score, so as to divide the video into time-non-repeating clips;

[0010] Based on the non-repeating time segments, video static information including visual information, audio information, subtitle information, emotional information, and time information, as well as video dynamic information including segment motion and sound changes are extracted and annotated, thereby realizing the generation of video segment labels.

[0011] Preferably, the scene change detection is used to preliminarily identify scene change points in the video and divide the video into segments with independent scenes, specifically:

[0012] Use feature extraction on the video to calculate the color histogram of each frame to describe the color distribution in the frame, use texture analysis methods to extract texture features of each frame, and detect the main objects in the video through the object detection algorithm;

[0013] The Euclidean distance is used to calculate the similarity of visual features between adjacent frames, and the similarity of visual features between adjacent frames obtained by scene change detection is recorded into a one-dimensional array Ascore of size T-1, where T is the number of video frames; the jth element of this array represents the scene change score of the j+1th frame of the original video. The lower the score, the higher the probability of a scene change point.

[0014] Preferably, the key frame detection is used to extract representative key frames from each scene segment, specifically:

[0015] One frame is selected every K frames, and the change of the picture content is used as the basis for selection, and the frame is selected as a candidate key frame; similar frames are grouped using K-means clustering, and the most representative frame is selected from each group, and the index I of these frames is recorded; Ascore[I] is set to ∞ to represent this frame as a key frame, because it cannot be a scene change point, so it will not be selected.

[0016] Preferably, the highlight detection algorithm is applied to identify the highlight area in the video frame, specifically:

[0017] Use binary search to detect highlights. Based on the deep learning-based highlight detection model QVHighlights, in each highlight detection process, a starting point Q and an ending point P are used as the model output results, which represent the starting and ending frames of the highlight area respectively. The two points Q and P are used to define the stable segments in the video for subsequent analysis.

[0018] Set hyperparameters w and σ, where w represents the weighted value, which is used to adjust the influence of the highlight detection segment on the visual feature similarity score Ascore, and σ represents the attenuation value after binary division iteration. This value controls the influence of the highlight segment in Ascore to gradually weaken with the increase of iteration times. Since there are a limited number of highlight segments in the video, the more iterations there are, the lower the probability of obtaining a highlight segment.

[0019] Perform weighted processing on the fragments detected by highlights and update the corresponding values ​​in Ascore, specifically:

[0020] Ascore[i]=Ascore[i]+σ j w

[0021] Where i∈[Q,P] represents each frame of highlight detection output, and j is the number of iterations. This weighting mechanism ensures that in each iteration, the impact of the detected highlight area on the visual feature similarity score is gradually attenuated, allowing new highlight information to be added in subsequent iterations;

[0022] The part [0, Q] before the highlight area of ​​the video clip and the part [P, T-1] after the highlight area of ​​the video clip are regarded as new video clips. The two new clips are re-detected for highlights to find other possible highlight areas. The newly detected highlight clips are weighted into Ascore again, and this is repeated M times, where M is a preset fixed value.

[0023] Preferably, the minimum point of the function of the search time and the scene change score is specifically:

[0024] Ascore is considered as a function of time t and scene change score, and local minimum points are detected by comparing the scores of adjacent frames. If:

[0025] Ascore[t-1]>Ascore[t] <Ascore[t+1],

[0026] Then t is a local minimum point, and:

[0027] t≠0andt≠T-1

[0028] Among them, t-1 and t+1 represent the indexes of the previous frame and the next frame of the current frame respectively. The first and last elements are directly excluded because they are the start frame and the end frame of the video.

[0029] The index t of the local minimum point that meets the conditions is stored in an array MIndices if:

[0030] Ascore[0] <Ascore[1],

[0031] Then 0 is a local minimum if:

[0032] Ascore[T-2]>Ascore[T-1],

[0033] Then T-1 is a local minimum point;

[0034] Determine the effective scene change point. The scene change point must be a local minimum point. The local minimum point is not necessarily a scene change point. It needs to pass the threshold test. If the score is not low enough to be a scene change point, set the threshold ∈ and find the minimum point t of the video. i So that:

[0035] Ascore[t i]<∈

[0036] t i ∈MIndices

[0037] The qualified t i It is recorded as one of the scene change points of the video and stored in an array MIndices, so as to divide the video into time-non-repeating segments.

[0038] Preferably, the extracting and marking of video static information including visual information, audio information, subtitle information, emotional information, and time information based on the time-non-repeating segments is specifically as follows:

[0039] Extract visual information from video clips, including person tags, object tags, event tags, and scene tags;

[0040] Extract character tags, annotate the main characters in the video through the target detection and recognition algorithm Yolov5, and generate character details, including gender, clothing, and skin color;

[0041] Extract item labels, identify and classify the main items or objects in the video, and use the object detection model Detectron2 to extract item labels;

[0042] Extract event labels, detect and mark the main events in the video, and mark the events through the action recognition model SlowFast or the event recognition model;

[0043] Extract scene labels, classify scenes according to visual content, and use the scene classification model SceneNet to analyze and label scene types;

[0044] Extract audio information from video clips, including sound source tags and music name tags;

[0045] Extract sound source labels, identify the sound sources in the audio, and use the sound classification model VGGish to annotate the sound sources;

[0046] Extract music name tags, classify and annotate background music, and use music information retrieval technology to identify music names;

[0047] Extract the speech content in the video and annotate it. Use the automatic speech recognition system Google Speech-to-Text API to transcribe the speech and generate labels. Generate different labels through the transcribed text of the conversation and the speaker separation results. Specifically, when the text is uninterrupted, indicating that one person is speaking continuously without anyone else interrupting, it is marked as a monologue; when there are multiple people speaking alternately in the text, it is marked as a dialogue; background noise or non-language elements need to be removed or marked as background sound;

[0048] Extract subtitle information from video clips, use natural language processing technology named entity recognition (NER) and part-of-speech tagging (POS) to extract nouns and entities, and use keyword extraction algorithms TF-IDF and TextRank to analyze and mark subtitle content;

[0049] Extract emotional information from video clips, perform sentiment analysis on the subtitles and audio in the video, determine the emotional state, and use the BERT model trained with the sentiment analysis training set to perform sentiment classification;

[0050] Extract the time information of the video clips, record the start and end time points of each clip, that is, the frame index in the scene transition point TransPoints, mark it with the timestamp information of the video, and based on the total length of the video and the relative position of the clip, mark the clip as being in the early, middle or late stage of the video.

[0051] Preferably, extracting and marking the video dynamic information including the segment action and sound changes according to the non-repeating segments in time is specifically as follows:

[0052] Generate action tokens for each video clip, use the action recognition model SlowFast to extract and mark the main actions or behaviors in the clip, convert the action features into token records for subsequent processing, analyze the visual changes in the video clip, and use the motion detection algorithm optical flow method to capture dynamic changes and generate corresponding dynamic information tokens;

[0053] Generate a sound change token for each audio clip, analyze and mark the changes in the audio, and use audio feature extraction technology to identify and record sound changes; analyze changes in speech content and use speech analysis models to generate speech change tokens;

[0054] Integrate the dynamic information tokens in the video clips and audio clips, integrate the video action tokens and the audio sound change tokens, generate a complete dynamic information description, and use the dynamic information integration algorithm to integrate the dynamic features of all clips to generate a global dynamic feature set that includes the video dynamic information of the clip action and the sound change.

[0055] Accordingly, the present invention also provides a video segment label generation system for scene change detection, comprising:

[0056] The video segmentation unit is used to segment video segments. First, scene change detection is used to preliminarily identify scene change points in the video. Then, the video is divided into segments with independent scenes. Key frame detection is used to extract representative key frames from each scene segment. A highlight detection algorithm is applied to identify highlight areas in the video frame. Finally, the minimum point of the function of time and scene change score is found, so as to divide the video into segments with non-repetitive time.

[0057] The information labeling unit is used to extract and label the video static information including visual information, audio information, subtitle information, emotional information, and time information, as well as the video dynamic information including segment actions and sound changes based on the non-repeating segments in time, so as to realize the generation of video segment labels.

[0058] The implementation of the present invention has the following beneficial effects:

[0059] The present invention can accurately segment videos into temporally non-repetitive segments through scene change detection, key frame extraction and highlight detection algorithms, thereby greatly improving the management and retrieval efficiency of video content. This fine segmentation can reduce redundant data and ensure that each segment represents an independent scene or event in the video.

[0060] By extracting visual, audio, subtitle and emotional information from videos and generating dynamic information tokens, the present invention can provide comprehensive labels and descriptions for video clips, which makes the analysis, retrieval and understanding of video content more comprehensive and supports a more intelligent content management system.

[0061] The static information tags (such as people, objects, events, etc.) and dynamic information tags (such as action tokens, sound change tokens) adopted in the present invention enable video content to be retrieved based on fine-grained information. This information richness improves the accuracy of the retrieval system, allowing users to find relevant video clips more quickly.

[0062] The present invention integrates the dynamic features of video and audio to generate a complete dynamic information description, which helps to conduct in-depth analysis of actions and sound changes in the video. This analysis capability is particularly important for application scenarios such as video surveillance and behavior recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 is a flow chart of a method for generating video segment labels for scene change detection according to an embodiment of the present invention;

[0064] Figure 2 It is a structural diagram of a video segment label generation system for scene change detection according to an embodiment of the present invention. DETAILED DESCRIPTION

[0065] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0066] Figure 1 FIG. 1 is a flow chart of a method for generating video segment labels for scene change detection according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0067] S1, video segmentation, first use scene change detection to preliminarily identify the scene change points in the video, then divide the video into segments with independent scenes, use key frame detection to extract representative key frames from each scene segment, and apply highlight detection algorithm to identify highlight areas in video frames, and finally find the minimum point of the function of time and scene change score, so as to divide the video into segments with non-repetitive time;

[0068] S2, based on the non-repeating segments in time, extract and annotate the video static information including visual information, audio information, subtitle information, emotional information, and time information, as well as the video dynamic information including segment actions and sound changes, so as to realize the generation of video segment labels.

[0069] Step S1 is as follows:

[0070] S1-1: Use scene change detection to divide the video into segments with independent scenes. Use feature extraction on the video to calculate the color histogram of each frame to describe the color distribution in the frame, use texture analysis method to extract texture features of each frame, and detect the main objects in the video through object detection algorithm;

[0071] The Euclidean distance is used to calculate the similarity of visual features between adjacent frames, and the similarity of visual features between adjacent frames obtained by scene change detection is recorded into a one-dimensional array Ascore of size T-1, where T is the number of video frames; the jth element of this array represents the scene change score of the j+1th frame of the original video. The lower the score, the higher the probability of a scene change point.

[0072] S1-2: Extract video key frames. Select one frame every K frames, use the change of the picture content as the basis for selection, and select the frame as a candidate key frame; use K-means clustering to group similar frames, and select the most representative frame from each group, and record the index I of these frames; set Ascore[I] to ∞ to represent this frame as a key frame, because it cannot be a scene change point, so it will not be selected.

[0073] S1-3: Identify highlight areas in video frames. Use binary search to detect highlights. Based on the deep learning-based highlight detection model QVHighlights, in each highlight detection process, a starting point Q and an ending point P are used as the model output results, which represent the starting and ending frames of the highlight area, respectively. Points Q and P are used to define stable segments in the video for subsequent analysis.

[0074] Set hyperparameters w and σ, where w represents the weighted value, which is used to adjust the influence of the highlight detection segment on the visual feature similarity score Ascore, and σ represents the attenuation value after binary division iteration. This value controls the influence of the highlight segment in Ascore to gradually weaken with the increase of iteration times. Since there are a limited number of highlight segments in the video, the more iterations there are, the lower the probability of obtaining a highlight segment.

[0075] Perform weighted processing on the fragments detected by highlights and update the corresponding values ​​in Ascore, specifically:

[0076] Ascore[i]=Ascore[i]+σ j w

[0077] Where i∈[Q,P] represents each frame of highlight detection output, and j is the number of iterations. This weighting mechanism ensures that in each iteration, the impact of the detected highlight area on the visual feature similarity score is gradually attenuated, allowing new highlight information to be added in subsequent iterations;

[0078] The part [0, Q] before the highlight area of ​​the video clip and the part [P, T-1] after the highlight area of ​​the video clip are regarded as new video clips. The two new clips are re-detected for highlights to find other possible highlight areas. The newly detected highlight clips are weighted into Ascore again, and this is repeated M times, where M is a preset fixed value.

[0079] S1-4: Find the minimum point of the function Ascore. Ascore is regarded as a function of time t and scene change score, and local minimum point detection is performed by comparing the scores of adjacent frames. If:

[0080] Ascore[t-1]>Ascore[t] <Ascore[t+1],

[0081] Then t is a local minimum point, and:

[0082] t≠0andt≠T-1

[0083] Among them, t-1 and t+1 represent the indexes of the previous frame and the next frame of the current frame respectively. The first and last elements are directly excluded because they are the start frame and the end frame of the video.

[0084] The index t of the local minimum point that meets the conditions is stored in an array MIndices if:

[0085] Ascore[0] <Ascore[1],

[0086] Then 0 is a local minimum if:

[0087] Ascore[T-2]>Ascore[T-1],

[0088] Then T-1 is a local minimum point;

[0089] Determine the effective scene change point. The scene change point must be a local minimum point. The local minimum point is not necessarily a scene change point. It needs to pass the threshold test. If the score is not low enough to be a scene change point, set the threshold ∈ and find the minimum point t of the video. i So that:

[0090] Ascore[t i ]<∈

[0091] t i ∈MIndices

[0092] The qualified t i It is recorded as one of the scene change points of the video and stored in an array MIndices, so as to divide the video into time-non-repeating segments.

[0093] Step S2 is as follows:

[0094] S2-1: Extract video static information, including visual information, audio information, subtitle information, emotional information, and time information, and annotate them.

[0095] S2-1-1: Extract visual information from video clips, including person tags, object tags, event tags, and scene tags;

[0096] Extract character tags, annotate the main characters in the video through the target detection and recognition algorithm Yolov5, and generate character details, including gender, clothing, and skin color;

[0097] Extract item labels, identify and classify the main items or objects in the video, and use the object detection model Detectron2 to extract item labels;

[0098] Extract event labels, detect and mark the main events in the video, and mark the events through the action recognition model SlowFast or the event recognition model;

[0099] Extract scene labels, classify scenes according to visual content, and use the scene classification model SceneNet to analyze and label scene types;

[0100] S2-1-2: extracting audio information from the video clip, including the sound source tag and the music name tag;

[0101] Extract sound source labels, identify the sound sources in the audio, and use the sound classification model VGGish to annotate the sound sources;

[0102] Extract music name tags, classify and annotate background music, and use music information retrieval technology to identify music names;

[0103] Extract the speech content in the video and annotate it. Use the automatic speech recognition system Google Speech-to-Text API to transcribe the speech and generate labels. Generate different labels through the transcribed text of the conversation and the speaker separation results. Specifically, when the text is uninterrupted, indicating that one person is speaking continuously without anyone else interrupting, it is marked as a monologue; when there are multiple people speaking alternately in the text, it is marked as a dialogue; background noise or non-language elements need to be removed or marked as background sound;

[0104] S2-1-3: Extract subtitle information from video clips, use natural language processing technology named entity recognition (NER) and part-of-speech tagging (POS) to extract nouns and entities, and use keyword extraction algorithms TF-IDF and TextRank to analyze and tag subtitle content;

[0105] S2-1-4: Extract emotional information from video clips, perform sentiment analysis on the subtitle content and audio in the video, determine the emotional state, and use the BERT model trained with the sentiment analysis training set to perform sentiment classification;

[0106] S2-1-5: Extract the time information of the video clips, record the start and end time points of each clip, that is, the frame index in the scene transition point TransPoints, mark it with the timestamp information of the video, and based on the total length of the video and the relative position of the clip, mark the clip as being in the early, middle or late stage of the video.

[0107] S2-2: Extract video dynamic information and annotate it, including segment motion and sound changes.

[0108] S2-2-1: Generate action tokens for each video clip, use the action recognition model SlowFast to extract and mark the main actions or behaviors in the clip, convert the action features into token records for subsequent processing, analyze the visual changes in the video clip, and use the motion detection algorithm optical flow method to capture dynamic changes and generate corresponding dynamic information tokens;

[0109] S2-2-2: Generate a sound change token for each audio clip, analyze the changes in the audio and mark them, use audio feature extraction technology to identify and record sound changes; analyze the changes in the speech content and generate a speech change token using the speech analysis model;

[0110] S2-2-3: Integrate the dynamic information tokens in the video clips and audio clips, integrate the video action tokens and the audio sound change tokens, generate a complete dynamic information description, and use the dynamic information integration algorithm to integrate the dynamic features of all clips to generate a global dynamic feature set that contains the video dynamic information of the clip action and the sound change.

[0111] Accordingly, the present invention also provides a video segment label generation system for scene change detection, such as Figure 2 As shown, including:

[0112] The video segmentation unit 1 is used to segment the video segments. First, scene change detection is used to preliminarily identify scene change points in the video. Then, the video is divided into segments with independent scenes. Key frame detection is used to extract representative key frames from each scene segment. A highlight detection algorithm is applied to identify highlight areas in the video frame. Finally, a minimum point of a function of time and scene change score is found, thereby dividing the video into segments with non-repetitive time.

[0113] The information labeling unit 2 is used to extract and label the video static information including visual information, audio information, subtitle information, emotional information, and time information, as well as the video dynamic information including segment actions and sound changes based on the non-repeating segments in time, so as to realize the generation of video segment labels.

[0114] Therefore, the present invention can accurately segment the video into temporally non-repeating segments through scene change detection, key frame extraction and highlight detection algorithms, thereby greatly improving the management and retrieval efficiency of video content. This fine segmentation can reduce redundant data and ensure that each segment represents an independent scene or event in the video; the present invention can provide comprehensive labels and descriptions for video segments by extracting visual, audio, subtitle and emotional information of the video and generating dynamic information tokens, which makes the analysis, retrieval and understanding of video content more comprehensive and supports a more intelligent content management system; the static information tags (such as people, objects, events, etc.) and dynamic information tags (such as action tokens, sound change tokens) used in the present invention enable video content to be retrieved based on fine-grained information. This information richness improves the accuracy of the retrieval system and enables users to find relevant video segments more quickly; the present invention integrates the dynamic features of video and audio to generate a complete dynamic information description, which helps to conduct in-depth analysis of actions and sound changes in the video. This analysis capability is particularly important for application scenarios such as video surveillance and behavior recognition.

[0115] The above is a detailed introduction to a method and system for generating video clip labels for scene change detection provided by an embodiment of the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea; at the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for generating video segment labels for scene change detection, characterized in that: The method comprises: To segment video clips, first use scene change detection to preliminarily identify scene change points in the video, then divide the video into clips with independent scenes, use key frame detection to extract representative key frames from each scene clip, and apply highlight detection algorithm to identify highlight areas in video frames, and finally find the minimum point of the function of time and scene change score, so as to divide the video into time-non-repeating clips; Based on the non-repeating time segments, video static information including visual information, audio information, subtitle information, emotional information, and time information, as well as video dynamic information including segment motion and sound changes are extracted and annotated, thereby realizing the generation of video segment labels.

2. The method for generating video segment labels for scene change detection according to claim 1, characterized in that: The scene change detection is used to preliminarily identify scene change points in the video and divide the video into segments with independent scenes, specifically: Use feature extraction on the video to calculate the color histogram of each frame to describe the color distribution in the frame, use texture analysis methods to extract texture features of each frame, and detect the main objects in the video through the object detection algorithm; The Euclidean distance is used to calculate the similarity of visual features between adjacent frames, and the similarity of visual features between adjacent frames obtained by scene change detection is recorded into a one-dimensional array Ascore of size T-1, where T is the number of video frames; the jth element of this array represents the scene change score of the j+1th frame of the original video. The lower the score, the higher the probability of a scene change point.

3. The method for generating video segment labels for scene change detection according to claim 2, characterized in that: The key frame detection is used to extract representative key frames from each scene segment, specifically: One frame is selected every K frames, and the change of the picture content is used as the basis for selection, and the frame is selected as a candidate key frame; similar frames are grouped using K-means clustering, and the most representative frame is selected from each group, and the index I of these frames is recorded; Ascore[I] is set to ∞ to represent this frame as a key frame, because it cannot be a scene change point, so it will not be selected.

4. The method for generating video segment labels for scene change detection according to claim 2, characterized in that: The highlight detection algorithm is applied to identify the highlight area in the video frame, specifically: Use binary search to detect highlights. Based on the deep learning-based highlight detection model QVHighlights, in each highlight detection process, a starting point Q and an ending point P are used as the model output results, which represent the starting and ending frames of the highlight area respectively. The two points Q and P are used to define the stable segments in the video for subsequent analysis. Set hyperparameters w and σ, where w represents the weighted value, which is used to adjust the influence of the highlight detection segment on the visual feature similarity score Ascore, and σ represents the attenuation value after binary division iteration. This value controls the influence of the highlight segment in Ascore to gradually weaken with the increase of iteration times. Since there are a limited number of highlight segments in the video, the more iterations there are, the lower the probability of obtaining a highlight segment. Perform weighted processing on the fragments detected by highlights and update the corresponding values ​​in Ascore, specifically: Ascore[i]=Ascore[i]+σ j w Where i∈[Q,P] represents each frame of highlight detection output, and j is the number of iterations. This weighting mechanism ensures that in each iteration, the impact of the detected highlight area on the visual feature similarity score is gradually attenuated, allowing new highlight information to be added in subsequent iterations; The part [0, Q] before the highlight area of ​​the video clip and the part [P, T-1] after the highlight area of ​​the video clip are regarded as new video clips. The two new clips are re-detected for highlights to find other possible highlight areas. The newly detected highlight clips are weighted into Ascore again, and this is repeated M times, where M is a preset fixed value.

5. The method for generating video segment labels for scene change detection according to claim 2, characterized in that: The minimum point of the function of the search time and the scene change score is specifically: Ascore is considered as a function of time t and scene change score, and local minimum points are detected by comparing the scores of adjacent frames. If: Ascore[t-1]>Ascore[t] <Ascore[t+1], Then t is a local minimum point, and: t≠0andt≠T-1 Among them, t-1 and t+1 represent the indexes of the previous frame and the next frame of the current frame respectively. The first and last elements are directly excluded because they are the start frame and the end frame of the video. The index t of the local minimum point that meets the conditions is stored in an array MIndices if: Ascore[0] <Ascore[1], Then 0 is a local minimum if: Ascore[T-2]>Ascore[T-1], Then T-1 is a local minimum point; Determine the effective scene change point. The scene change point must be a local minimum point. The local minimum point is not necessarily a scene change point. It needs to pass the threshold test. If the score is not low enough to be a scene change point, set the threshold ∈ and find the minimum point t of the video. i So that: Ascore[t i ]<∈ t i ∈MIndices The qualified t i It is recorded as one of the scene change points of the video and stored in an array MIndices, so as to divide the video into time-non-repeating segments.

6. The method for generating video segment labels for scene change detection according to claim 1, characterized in that: The extracting and marking of video static information including visual information, audio information, subtitle information, emotional information, and time information according to the time-non-repeating segments is specifically as follows: Extract visual information from video clips, including person tags, object tags, event tags, and scene tags; Extract character tags, annotate the main characters in the video through the target detection and recognition algorithm Yolov5, and generate character details, including gender, clothing, and skin color; Extract item labels, identify and classify the main items or objects in the video, and use the object detection model Detectron2 to extract item labels; Extract event labels, detect and mark the main events in the video, and mark the events through the action recognition model SlowFast or the event recognition model; Extract scene labels, classify scenes according to visual content, and use the scene classification model SceneNet to analyze and label scene types; Extract audio information from video clips, including sound source tags and music name tags; Extract sound source labels, identify the sound sources in the audio, and use the sound classification model VGGish to annotate the sound sources; Extract music name tags, classify and annotate background music, and use music information retrieval technology to identify music names; Extract the speech content in the video and annotate it. Use the automatic speech recognition system Google Speech-to-Text API to transcribe the speech and generate labels. Generate different labels through the transcribed text of the conversation and the speaker separation results. Specifically, when the text is uninterrupted, indicating that one person is speaking continuously without anyone else interrupting, it is marked as a monologue; when there are multiple people speaking alternately in the text, it is marked as a dialogue; background noise or non-language elements need to be removed or marked as background sound; Extract subtitle information from video clips, use natural language processing technology named entity recognition (NER) and part-of-speech tagging (POS) to extract nouns and entities, and use keyword extraction algorithms TF-IDF and TextRank to analyze and mark subtitle content; Extract emotional information from video clips, perform sentiment analysis on the subtitles and audio in the video, determine the emotional state, and use the BERT model trained with the sentiment analysis training set to perform sentiment classification; Extract the time information of the video clips, record the start and end time points of each clip, that is, the frame index in the scene transition point TransPoints, mark it with the timestamp information of the video, and based on the total length of the video and the relative position of the clip, mark the clip as being in the early, middle or late stage of the video.

7. The method for generating video segment labels for scene change detection according to claim 1, characterized in that: The extracting and marking of video dynamic information including segment actions and sound changes based on the non-repeating segments in time is specifically as follows: Generate action tokens for each video clip, use the action recognition model SlowFast to extract and mark the main actions or behaviors in the clip, convert the action features into token records for subsequent processing, analyze the visual changes in the video clip, and use the motion detection algorithm optical flow method to capture dynamic changes and generate corresponding dynamic information tokens; Generate a sound change token for each audio clip, analyze and mark the changes in the audio, and use audio feature extraction technology to identify and record sound changes; analyze changes in speech content and use speech analysis models to generate speech change tokens; Integrate the dynamic information tokens in the video clips and audio clips, integrate the video action tokens and the audio sound change tokens, generate a complete dynamic information description, and use the dynamic information integration algorithm to integrate the dynamic features of all clips to generate a global dynamic feature set that includes the video dynamic information of the clip action and the sound change.

8. A video segment label generation system for scene change detection, characterized in that: The system comprises: The video segmentation unit is used to segment video segments. First, scene change detection is used to preliminarily identify scene change points in the video. Then, the video is divided into segments with independent scenes. Key frame detection is used to extract representative key frames from each scene segment. A highlight detection algorithm is applied to identify highlight areas in the video frame. Finally, the minimum point of the function of time and scene change score is found, so as to divide the video into segments with non-repetitive time. The information labeling unit is used to extract and label the video static information including visual information, audio information, subtitle information, emotional information, and time information, as well as the video dynamic information including segment actions and sound changes based on the non-repeating segments in time, so as to realize the generation of video segment labels.

9. The video segment label generation system for scene change detection according to claim 8, characterized in that: The video segmentation unit is used to use scene change detection to preliminarily identify scene change points in the video and divide the video into segments with independent scenes, specifically: Use feature extraction on the video to calculate the color histogram of each frame to describe the color distribution in the frame, use texture analysis methods to extract texture features of each frame, and detect the main objects in the video through the object detection algorithm; The Euclidean distance is used to calculate the similarity of visual features between adjacent frames, and the similarity of visual features between adjacent frames obtained by scene change detection is recorded into a one-dimensional array Ascore of size T-1, where T is the number of video frames; the jth element of this array represents the scene change score of the j+1th frame of the original video. The lower the score, the higher the probability of a scene change point.

10. The video segment label generation system for scene change detection according to claim 9, characterized in that: The video segmentation unit is used to extract representative key frames from each scene segment by using key frame detection, specifically: One frame is selected every K frames, and the change of the picture content is used as the basis for selection, and the frame is selected as a candidate key frame; similar frames are grouped using K-means clustering, and the most representative frame is selected from each group, and the index I of these frames is recorded; Ascore[I] is set to ∞ to represent this frame as a key frame, because it cannot be a scene change point, so it will not be selected.

11. The video segment label generation system for scene change detection according to claim 9, characterized in that: The video segmentation unit is used to apply a highlight detection algorithm to identify highlight areas in the video frame, specifically: Use binary search to detect highlights. Based on the deep learning-based highlight detection model QVHighlights, in each highlight detection process, a starting point Q and an ending point P are used as the model output results, which represent the starting and ending frames of the highlight area respectively. The two points Q and P are used to define the stable segments in the video for subsequent analysis. Set hyperparameters w and σ, where w represents the weighted value, which is used to adjust the influence of the highlight detection segment on the visual feature similarity score Ascore, and σ represents the attenuation value after binary division iteration. This value controls the influence of the highlight segment in Ascore to gradually weaken with the increase of iteration times. Since there are a limited number of highlight segments in the video, the more iterations there are, the lower the probability of obtaining a highlight segment. Perform weighted processing on the fragments detected by highlights and update the corresponding values ​​in Ascore, specifically: Ascore[i]=Ascore[i]+σ j w Where i∈[Q,P] represents each frame of highlight detection output, and j is the number of iterations. This weighting mechanism ensures that in each iteration, the impact of the detected highlight area on the visual feature similarity score is gradually attenuated, allowing new highlight information to be added in subsequent iterations; The part [0, Q] before the highlight area of ​​the video clip and the part [P, T-1] after the highlight area of ​​the video clip are regarded as new video clips. The two new clips are re-detected for highlights to find other possible highlight areas. The newly detected highlight clips are weighted into Ascore again, and this is repeated M times, where M is a preset fixed value.

12. The video segment label generation system for scene change detection according to claim 9, characterized in that: The minimum point of the function of the search time and the scene change score is specifically: Ascore is considered as a function of time t and scene change score, and local minimum points are detected by comparing the scores of adjacent frames. If: Ascore[t-1]>Ascore[t] <Ascore[t+1], Then t is a local minimum point, and: t≠0andt≠T-1 Among them, t-1 and t+1 represent the indexes of the previous frame and the next frame of the current frame respectively. The first and last elements are directly excluded because they are the start frame and the end frame of the video. The index t of the local minimum point that meets the conditions is stored in an array MIndices if: Ascore[0] <Ascore[1], Then 0 is a local minimum if: Ascore[T-2]>Ascore[T-1], Then T-1 is a local minimum point; Determine the effective scene change point. The scene change point must be a local minimum point. The local minimum point is not necessarily a scene change point. It needs to pass the threshold test. If the score is not low enough to be a scene change point, set the threshold ∈ and find the minimum point t of the video. i So that: Ascore[t i ]<∈ t i ∈MIndices The qualified t i It is recorded as one of the scene change points of the video and stored in an array MIndices, so as to divide the video into time-non-repeating segments.

13. The video segment label generation system for scene change detection according to claim 8, characterized in that: The information annotation unit is used to extract and annotate video static information including visual information, audio information, subtitle information, emotional information, and time information according to the video clip, specifically: Extract visual information from video clips, including person tags, object tags, event tags, and scene tags; Extract character tags, annotate the main characters in the video through the target detection and recognition algorithm Yolov5, and generate character details, including gender, clothing, and skin color; Extract item labels, identify and classify the main items or objects in the video, and use the object detection model Detectron2 to extract item labels; Extract event labels, detect and mark the main events in the video, and mark the events through the action recognition model SlowFast or the event recognition model; Extract scene labels, classify scenes according to visual content, and use the scene classification model SceneNet to analyze and label scene types; Extract audio information from video clips, including sound source tags and music name tags; Extract sound source labels, identify the sound sources in the audio, and use the sound classification model VGGish to annotate the sound sources; Extract music name tags, classify and annotate background music, and use music information retrieval technology to identify music names; Extract the speech content in the video and annotate it. Use the automatic speech recognition system Google Speech-to-Text API to transcribe the speech and generate labels. Generate different labels through the transcribed text of the conversation and the speaker separation results. Specifically, when the text is uninterrupted, indicating that one person is speaking continuously without anyone else interrupting, it is marked as a monologue; when there are multiple people speaking alternately in the text, it is marked as a dialogue; background noise or non-language elements need to be removed or marked as background sound; Extract subtitle information from video clips, use natural language processing technology named entity recognition (NER) and part-of-speech tagging (POS) to extract nouns and entities, and use keyword extraction algorithms TF-IDF and TextRank to analyze and mark subtitle content; Extract emotional information from video clips, perform sentiment analysis on the subtitles and audio in the video, determine the emotional state, and use the BERT model trained with the sentiment analysis training set to perform sentiment classification; Extract the time information of the video clips, record the start and end time points of each clip, that is, the frame index in the scene transition point TransPoints, mark it with the timestamp information of the video, and based on the total length of the video and the relative position of the clip, mark the clip as being in the early, middle or late stage of the video.

14. The video segment label generation system for scene change detection according to claim 8, characterized in that: The information annotation unit is used to extract and annotate video dynamic information including segment actions and sound changes according to the video segment, specifically: Generate action tokens for each video clip, use the action recognition model SlowFast to extract and mark the main actions or behaviors in the clip, convert the action features into token records for subsequent processing, analyze the visual changes in the video clip, and use the motion detection algorithm optical flow method to capture dynamic changes and generate corresponding dynamic information tokens; Generate a sound change token for each audio clip, analyze and mark the changes in the audio, and use audio feature extraction technology to identify and record sound changes; analyze changes in speech content and use speech analysis models to generate speech change tokens; Integrate the dynamic information tokens in the video clips and audio clips, integrate the video action tokens and the audio sound change tokens, generate a complete dynamic information description, and use the dynamic information integration algorithm to integrate the dynamic features of all clips to generate a global dynamic feature set that includes the video dynamic information of the clip action and the sound change.

Images