A video scene boundary detection method based on multi-modal decision fusion

Abstract

The present application relates to a kind of video scene boundary detection method based on multi-modal decision fusion, belong to video scene boundary detection technical field, solve the problem of low accuracy rate of scene boundary detection in prior art while adapting to multiple contents.The method comprises the following steps: carrying out shot boundary detection to the video to be detected, obtain each shot in the video to be detected;In each mode, judge whether each shot is transition boundary, obtain the boundary decision of each mode;Based on the boundary decision of each mode, construct multi-modal fuzzy decision space, based on multi-modal fuzzy decision space, carry out scene boundary decision and obtain the scene boundary of the video to be detected.It realizes more accurate scene boundary detection while facing multiple contents.

Description

Technical Field

[0001] This invention relates to the field of video scene boundary detection technology, and in particular to a video scene boundary detection method based on multimodal decision fusion. Background Technology

[0002] The development of communication technology and advancements in filming equipment have brought about a dramatic change in the media information dissemination landscape, with the mainstream form of online information dissemination evolving from text to video. Conventional video content creation can no longer meet people's demands for video, and secondary content creation based on existing videos is receiving increasing attention. Video content reproduction requires segmenting videos into small-granular segments such as shots and scenes. This paper implements video segmentation for news broadcasts and variety shows using a video scene boundary detection system based on multimodal decision fusion.

[0003] The earliest definition of a scene originates from cinematography. A scene is defined as a sequence of images unified with time and space, composed of a series of shots taken from different angles, usually filmed within a single time period. As an independent semantic unit, a scene lies between a shot and a sequence; it is larger than a shot but smaller than a video sequence. A scene is composed of a series of shots, and the story presented by these shots is semantically coherent, describing events occurring at the same time and place, or multiple parallel events intertwined. In the field of news videos, scenes are often described as news story units, meaning a complete news story constitutes a scene. For variety show videos, the filming techniques and program content vary greatly depending on the type of variety show. This article focuses on videos where the host opens the show and then the actors present the content; in this case, a scene can be described as a complete program unit.

[0004] Video scene boundary detection can be divided into two categories: unimodal video scene boundary detection algorithms and multimodal video scene boundary detection algorithms. Unimodal video scene boundary detection algorithms extract visual modal features as input and utilize the visual characteristics of keyframes in the shot combined with the scene's composition pattern to perform video scene boundary detection.

[0005] Multimodal video scene boundary detection can learn more complete feature representations and is more content-adaptive. In scene boundary detection tasks that simultaneously target multiple content types, such as news reports and variety shows, the correlation between text semantics and visual features is relatively weak, making decision fusion a more suitable approach than feature fusion. However, in news reports and variety show videos, the single-modal semantic transformation boundaries of available video, text, or audio data each have different positional offsets from the real scene boundaries, affecting the accuracy of decision-making and resulting in low accuracy. Summary of the Invention

[0006] Based on the above analysis, the embodiments of the present invention aim to provide a video scene boundary detection method based on multimodal decision fusion, in order to solve the problem of low accuracy of existing scene boundary detection methods that simultaneously adapt to multiple contents.

[0007] On one hand, embodiments of the present invention provide a video scene boundary detection method based on multimodal decision fusion, comprising the following steps:

[0008] Perform shot boundary detection on the video to be inspected to obtain each shot in the video to be inspected;

[0009] In each modality, determine whether each shot is a transition boundary to obtain the boundary decision for each modality;

[0010] A multimodal fuzzy decision space is constructed based on the boundary decisions of each modality, and the scene boundary of the video to be detected is obtained by performing scene boundary decisions based on the multimodal fuzzy decision space.

[0011] Based on further improvements to the above method, a multimodal fuzzy decision space is constructed based on the boundary decisions of each mode, including:

[0012] Calculate the boundary ambiguity for each mode;

[0013] The decision space after fuzzing each mode is obtained based on the boundary fuzziness of each mode;

[0014] A multimodal fuzzy decision space is constructed based on the decision space after each modality is fuzzified.

[0015] Based on the above method, a further improvement is made, and the boundary ambiguity of each mode is calculated using the following procedure:

[0016] Construct a scene boundary dataset; each sample in the scene boundary dataset includes a shot of each video, as well as a decision on whether each shot is a unimodal semantic boundary in each modality and a label on whether it is a scene boundary;

[0017] Calculate the total number of scene boundaries with different offsets for each sample in the scene boundary dataset in each modality;

[0018] The boundary ambiguity for each mode is calculated based on the total number of scene boundaries with different offsets for each sample in the scene boundary dataset for each mode.

[0019] Based on the above method, a further improvement is made, and the following formula is used to calculate the total number of scene boundaries with different offsets for each sample in each modality in the scene boundary dataset:

[0020]

[0021] Among them, D m(i) represents the decision result of the i-th shot in the boundary decision of the sample in modality m, d represents the offset length, G(i+d) represents the label of whether the i+d-th shot is the scene boundary, and k represents the boundary decision D of the sample in modality m. m The length of N m (d) represents the total number of scene boundaries with an offset length of d from the scene boundary in the boundary decision of the sample in the m-mode m.

[0022] Based on the further improvement of the above method, the boundary ambiguity of each mode is calculated using the following formula, according to the total number of scene boundaries with different offsets for each sample in the scene boundary dataset in each mode:

[0023]

[0024] in, N represents the total number of boundary scenes with an offset length of d for the j-th sample in mode m, and N represents the total number of samples. Let P represent the total number of boundary scenes with an offset length of l for the j-th sample in modality m, where K represents the maximum offset length. m (d) represents the boundary ambiguity when the offset length is d under mode m.

[0025] Based on a further improvement to the above method, the decision space after fuzzification for each mode is obtained according to the boundary fuzziness of each mode, including:

[0026] For the transition boundary in the boundary decision of each mode, the formula is used. Calculate the fuzzy values ​​to obtain the fuzzy decision space;

[0027] Among them, P m (d) represents the fuzziness of the boundary decision in mode m when the offset length is d, where d represents the offset length, D m (id) represents the decision result of the id-th shot in the boundary decision under modality m. Let k represent the decision space after blurring the decision result of the i-th shot in the boundary decision under modality m, and let k represent the boundary decision D under modality m. m The length.

[0028] Based on a further improvement of the above method, the multimodal mode includes a text modality. In the text modality, determining whether each shot is a transition boundary yields the boundary decision for the text modality, including:

[0029] The title sequence is obtained by detecting the title in the keyframes of each shot;

[0030] Based on the similarity of adjacent thematic subtitles in the thematic subtitle sequence, we determine whether each shot is a transition boundary, thus obtaining the boundary decision of the text modality.

[0031] Based on the above method, a further improvement is made, and the similarity between two adjacent topic captions is calculated using the following formula:

[0032]

[0033] If the similarity d text If (i, i+1) is less than the first threshold, then the shot corresponding to the i-th theme subtitle is the transition boundary;

[0034] Among them, T i T represents the i-th theme subtitle in the theme subtitle sequence. i+1 T represents the (i+1)th theme subtitle in the theme subtitle sequence. i ∩T i+1 T represents i and T i+1 The intersection of T i ∪T i+1 T represents i and T i+1 The union of the elements, where crad(·) represents the function to calculate the number of elements.

[0035] Based on a further improvement of the above method, the multimodal mode includes a video mode. In the video mode, determining whether each shot is a transition boundary yields the boundary decision for the video mode, including:

[0036] Detect facial features in keyframes of each shot;

[0037] If the similarity between the detected facial feature and the facial feature of the key person exceeds the second threshold, then the shot corresponding to the detected facial feature is the conversion boundary; thus, the boundary decision of the video modality is obtained.

[0038] Based on a further improvement of the above method, the multimodal mode includes an audio mode. In the audio mode, determining whether each shot is a transition boundary yields the boundary decision for the audio mode, including:

[0039] The audio signal of the video to be detected is extracted, and the silent region in the audio signal is obtained based on the root mean square energy.

[0040] The shot corresponding to the start time of the silent zone is the transition boundary; the boundary decision of the audio modality is obtained.

[0041] Compared with existing technologies, the video scene boundary detection method based on multimodal decision fusion provided in this invention first performs shot detection on the video to be detected to obtain each shot in the video to be detected. Then, it determines whether each shot is a transition boundary in each modality, thereby obtaining the boundary decision result in each modality. By constructing a multimodal fuzzy decision space based on the boundary decision of each modality, and then performing decision fusion judgment based on the multimodal decision space, the scene boundary in the video to be detected can be obtained quickly and accurately, which improves the accuracy of scene boundary detection. It is also adaptable to news and variety shows, and has good generalization performance in some sports events.

[0042] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0043] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0044] Figure 1 This is a flowchart of a video scene boundary detection method based on multimodal decision fusion according to an embodiment of the present invention;

[0045] Figure 2 This is a schematic diagram illustrating annotations in an embodiment of the present invention;

[0046] Figure 3 This is a schematic diagram of subtitle types according to an embodiment of the present invention. Detailed Implementation

[0047] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0048] Existing video scene segmentation techniques are divided into two categories: time-based segmentation and shot-based segmentation. Time-based segmentation has a precision of seconds, which can lead to a single shot being divided into two scenes. Shot-based segmentation considers the hierarchical structure of the video and is suitable for multi-camera edited videos. This invention addresses scene boundary detection tasks in multi-camera videos such as news broadcasts and variety shows, where textual semantics and visual features exhibit weak correlation, making decision fusion a more suitable approach than feature fusion. Therefore, this invention employs shot-based video scene segmentation, where the scene boundary set is a subset of the shot boundary set. Scene segmentation is performed by locating the scene boundaries.

[0049] Multi-camera shots of the presenter and the absence of a presenter at the beginning of a scene present challenges for visual modality-based scene boundary detection. Furthermore, the positional offset between subtitle transition boundaries and scene boundaries, with varying offset lengths, hinders text-based single-modality video scene boundary detection. The presence of audio commas and background noise makes it difficult to locate video scene boundaries solely based on audio comma boundaries. In other words, there is a discrepancy between single-modality semantic boundaries and scene boundaries, resulting in a single-modality semantic boundary set that only includes a portion of the scene boundaries. Therefore, maximizing the number of correctly identified scene boundaries by leveraging the complementary relationships between different modal boundaries and improving the accuracy of scene boundary detection is a pressing issue that needs to be addressed.

[0050] A specific embodiment of the present invention discloses a video scene boundary detection method based on multimodal decision fusion, such as... Figure 1 As shown, it includes the following steps:

[0051] S1. Perform lens boundary detection on the video to be detected to obtain each lens in the video to be detected;

[0052] S2. Determine whether each shot is a transition boundary in each modality to obtain the boundary decision for each modality;

[0053] S3. Construct a multimodal fuzzy decision space based on the boundary decisions of each modality, and obtain the scene boundary of the video to be detected by performing scene boundary decisions based on the multimodal fuzzy decision space.

[0054] It should be noted that detecting each shot in the video to be detected means detecting the start and end frame numbers of each shot in the video. The end frame of each shot is taken as the shot boundary frame, and scene boundary frames are a subset of shot boundary frames. Determining whether each shot is a transition boundary is equivalent to determining whether each shot boundary frame is a scene boundary frame.

[0055] Compared with existing technologies, the video scene boundary detection method based on multimodal decision fusion provided in this invention first determines the shots in the video to be detected, and then determines whether each shot is a transition boundary in each modality, thereby obtaining the boundary decision result in each modality. A multimodal fuzzy decision space is constructed by the boundary decisions of each modality, and then decision fusion is performed based on the multimodal decision space to quickly and accurately obtain the scene boundaries in the video to be detected, improving the accuracy of scene boundary detection for multiple types of content simultaneously. This technology is also adaptable to news and variety shows, and has good generalization performance in some sports events.

[0056] In implementation, by analyzing the scene structure of news reports and variety shows, and combining existing scene boundary determination methods, scene annotation criteria for news reports and variety show videos are first formulated, and a data cluster, Scene-NSG, is created. This dataset includes a main dataset, Scene-NSG-M, which contains video scene boundaries, and a target recognition subset, Scene-NSG-OD, which focuses on key figures and subtitle recognition. Scene-NSG-M annotates news reports and variety show videos from multiple programs on different channels; each sample in the Scene-NSG-OD dataset is labeled with the target category and target location. This invention is based on the Scene-NSG data cluster, and therefore is adaptable to both news and variety shows.

[0057] There are two methods for annotating video scene datasets. The first method uses frames as markers to annotate shot boundaries and scene boundaries separately. The second method uses shots as markers, annotating the shots corresponding to scene transitions as scene boundaries, i.e., the boundary frames of shots are the scene boundary frames. The scene dataset annotation method of this invention is the second method. For a given video, this invention first performs shot boundary detection, and then determines whether each shot is an audio comma boundary, a key person transition boundary, a subtitle transition boundary, and a scene boundary.

[0058] It should be noted that, yes, in news reports and variety shows, the key person is the host, while in sports event videos, the key person can be the announcer or the person indicating the transition.

[0059] In implementation, the labeled sample of a sample in the Scene-NSG-M dataset is as follows: Figure 2 As shown, if the current shot contains a silent area (a silent area lasting longer than 1 second is considered a silent area), then the shot is an audio comma boundary, and the duration of the silent area in the shot is used as the annotation content of the shot in the audio modality; shots that do not contain silent areas are annotated as 0 in the audio modality, resulting in the audio comma boundary set D. aIf the current shot shows a change in the host or the host's camera position compared to the next shot, then the current shot is considered a host transition boundary, and the current shot is labeled as 1 in the video modality. If there is no change in the host or the host's camera position, then the current shot is labeled as 0 in the video modality. The labeling results in the host transition boundary set D. v If the content of the title tag identified in the current shot changes, then the current shot is considered a title transition boundary, and the current shot is labeled as 1 in text mode. Otherwise, the current shot is labeled as 0 in text mode, resulting in the title transition boundary set D. t In the process of labeling scene boundaries using shots as markers, the last shot of each scene is labeled as 1 as the scene boundary, and the other shots within the scene are labeled as 0, thus obtaining the true value set G of the scene boundaries.

[0060] In implementation, the multi-type shot boundary detection (MSBD) algorithm can be used to detect shot boundaries in the video. That is, it detects whether each video frame belongs to the same shot as the previous frame. If not, the current video frame is the starting frame of a new shot, and the previous frame is the ending frame of the previous shot; the ending frame of each shot is used as the boundary frame of that shot.

[0061] In practice, for accurate detection, the keyframes for each shot are first determined based on its start and end frames. For each shot in the video, a video frame between its start and end frames is taken as the keyframe for that shot. If the video to be detected has k shots, there are k corresponding keyframes, which are used for subsequent single-modal feature analysis.

[0062] Then, in each modality, it is determined whether each shot is a transition boundary, thus obtaining the boundary decision for each modality. In implementation, in order to integrate more heterogeneous data and improve the accuracy and generalization ability of scene boundary detection, the multimodal nature of this invention includes text modality, video modality, and audio modality.

[0063] In news reports, variety shows, and other videos, the theme subtitles for the same scene remain almost unchanged, while the theme subtitles for different scenes are usually different. Therefore, identifying theme subtitles helps to accurately determine scene boundaries and improve the accuracy of scene boundary judgment.

[0064] In text modality, determine whether each shot is a transition boundary to obtain text modality boundary decisions, including:

[0065] S201. Detect the theme subtitles in the keyframes of each shot to obtain the theme subtitle sequence;

[0066] S202. Based on the similarity of adjacent theme subtitles in the theme subtitle sequence, determine whether each shot is a transition boundary to obtain the boundary decision of the subtitle modality.

[0067] In practice, the subtitles in the video include title subtitles, announcement subtitles, and audio subtitles, such as... Figure 3 As shown. Figure 3 The left side shows an example of audio subtitles, the middle side shows an example of title subtitles, and the right side shows an example of announcement subtitles. Title subtitles and announcement subtitles are collectively referred to as theme subtitles. Audio subtitles display the subtitles corresponding to the audio in the video, while title subtitles and announcement subtitles are used to display thematic information about the scene.

[0068] During implementation, keyframes of the shot are input into the target detection model, which can detect the subtitles and their types contained in the keyframes.

[0069] After detecting the theme subtitle, OCR recognition technology is used to extract the text contained in the theme subtitle area, and the text difference calculation is used to find the shot where the theme subtitle changes, and this shot is used as the subtitle change boundary.

[0070] For example, the sequence of title subtitles extracted from k keyframes is w = {w1, w2, w3, ... w}. n}, w i It contains two parameters w i =[nt i ,T i ], where nt i T is the shot number corresponding to the keyframe where the i-th theme subtitle is located. i It is the set of strings for the i-th theme subtitle.

[0071] In news reports and variety show videos, the subtitles for the same scene rarely change. Therefore, the intersection-union ratio of string sets can be used as a measure of text difference d. text This allows us to determine whether a title change has occurred. Specifically, the similarity between two adjacent title tags is calculated using the following formula:

[0072]

[0073] If the similarity d text If (i, i+1) is less than the first threshold, then the shot corresponding to the i-th theme subtitle is the transition boundary;

[0074] Among them, T i T represents the i-th theme subtitle in the theme subtitle sequence. i+1 T represents the (i+1)th theme subtitle in the theme subtitle sequence. i ∩T i+1 T represents i and T i+1 The intersection of Ti ∪T i+1 T represents i and T i+1 The union of the elements, where crad(·) represents the function to calculate the number of elements.

[0075] d text ∈[0,1], the more similar the theme subtitles, the better d text The larger the value, the more likely it is that when two title tags do not have the same characters, d text The value is 0 when the two theme subtitles are exactly the same. text The value is 1.

[0076] When d text Less than the first threshold T text When d, it is considered that a subtitle change has occurred. text Greater than or equal to the first threshold T text At that time, it is assumed that no subtitle conversion has occurred, thus obtaining the boundary decision D of the text modality. t During implementation, T text Set to 0.6. It should be noted that the boundary decision of the text modality is a sequence of length k, where the i-th element is either 1 or 0. When the i-th shot is a transition boundary, the i-th element of the boundary decision of the text modality is 1, otherwise it is 0.

[0077] When transitioning between scenes, key figures usually announce the transitions, so the accuracy of scene boundary detection can be improved by detecting key figures.

[0078] Specifically, in the video modality, determining whether each shot is a transition boundary yields the video modality boundary decision, including:

[0079] S211. Detect facial features in the keyframes of each shot;

[0080] S212. If the similarity between the detected facial feature and the facial feature of the key person exceeds the second threshold, then the shot corresponding to the detected facial feature is the conversion boundary; thus, the boundary decision of the video modality is obtained.

[0081] During implementation, keyframes from the camera are input into the target detection model, which can detect faces contained in the keyframes and whether they are key people.

[0082] In implementation, the object detection model in steps S201 and S211 can be the same model. A pre-trained YOLOv5 model on the COCO dataset can be used, and then fine-tuned on a self-constructed dataset, Scene-NSG-OD, to obtain a trained object detection model. This object detection model is then used to detect subtitles and faces in keyframes.

[0083] In implementation, the category data of the YOLOv5 prediction head is modified to 5 to detect 5 types of targets: voice captions, title captions, program captions, presenter, and others. The weights of the backbone and neck parts are fine-tuned to obtain the trained target detection model.

[0084] In implementation, keyframes of the shot are input into the object detection model to obtain the host's face contained within the keyframes. However, there are instances where other people are mistakenly identified as the host, leading to errors in transition boundary detection. Therefore, the facial features of the host identified by the object detection model are extracted, and then similarity is calculated with the facial features of the actual host. Based on the similarity, it is determined whether the corresponding shot is a transition boundary. In implementation, MobileFaceNet can be used to extract the features of the identified facial regions.

[0085] For example, if three presenters are detected in a keyframe of a shot, MobileFaceNet is used to extract the features of the face regions of the three detected presenters. Then, the similarity is calculated between these features and the facial features of the real presenters. If the similarity between the facial features of any detected presenter and the facial features of a real presenter is greater than a second threshold, then the shot corresponding to that keyframe is considered a transformation boundary; otherwise, the shot is not a transformation boundary. This yields the boundary decision D for the video modality. v During implementation, the second threshold T host The value can be set to 0.8. It should be noted that the boundary decision of the video modality is a sequence of length k, where the i-th element is either 1 or 0. When the i-th shot is a transition boundary, the i-th element of the boundary decision of the video modality is 1, otherwise it is 0.

[0086] When implementing, a formula can be used. Calculate the similarity, where f i v This represents the facial features of the i-th detected host. This represents the facial features of the actual a-th presenter.

[0087] During scene transitions, there are often audio gaps, or silent areas. Therefore, these silent areas can be used to identify scene transition boundaries. Specifically, determining whether each shot represents a transition boundary in the audio modality involves the following boundary decisions:

[0088] S231. Extract the audio signal from the video to be detected, and obtain the silent area in the audio signal based on the root mean square energy.

[0089] S232, the shot corresponding to the start time of the silent zone is the transition boundary; obtain the boundary decision of the audio mode.

[0090] During implementation, the audio signal is first extracted from the video to be detected, and then the audio comma, i.e. the silence zone, is found by the magnitude of the root mean square (RMS) energy of the audio.

[0091] Specifically, the silence region in the audio signal is obtained based on the root mean square energy, including:

[0092] S2311. Use a sliding window to extract signal segments of the audio signal and calculate the root mean square energy of each signal segment;

[0093] S2312. If the root mean square energy of a signal segment is less than the third threshold, then the signal segment is a silent segment.

[0094] S2313. If there are continuous silence segments, and the time interval between the start time of the first silence segment and the start time of the last silence segment in the continuous silence segments is greater than the fourth threshold, then the start time of the first silence segment in the continuous silence segments is taken as the start time of the silence zone, and the start time of the last silence segment is taken as the end time of the silence zone, thus obtaining a silence zone.

[0095] In implementation, a sliding window is used to slide across the audio signal from beginning to end. The length of the sliding window is L signal points. The signal segment within each sliding window is extracted, and the root mean square energy is calculated using the following formula:

[0096]

[0097] Among them, y a The value represents the amplitude of the a-th signal point in the signal segment, k is the digital audio sampling rate, and h is the total audio duration.

[0098] If the root mean square energy of a signal segment is less than the third threshold, the signal segment is considered a silent segment. The third threshold is the maximum energy value of a preset silent zone. In practice, the third threshold T... audio It can be -16dB.

[0099] For each silent segment, its start time is recorded. If there are two or more consecutive silent segments, and the time interval between the start time of the first silent segment and the start time of the last silent segment is greater than a fourth threshold, then a silent zone is formed. The start time of the first silent segment in the consecutive silent segments is the start time of a silent zone, and the start time of the last silent segment is the end time of the silent zone. In practice, the fourth threshold can be set to 1 second.

[0100] The shot corresponding to the start time of the silent zone is the transition boundary. For example, if the start time of a silent zone is at second t, and the frame rate of the video to be detected is f, then the frame number N corresponding to the start time of the silent zone is...f = t × f.

[0101] If the frame number N f If the i-th shot begins after its first frame (including the first frame of the i-th shot) and begins before the (i+1)-th shot, then the i-th shot is the transition boundary. This yields the boundary decision D for the audio modality. a It should be noted that the boundary decision of the audio modality is a sequence of length k, where the i-th element is 0 or a number greater than 0. When the i-th shot is a transition boundary, the i-th element of the audio modality boundary decision is the length of the corresponding silent zone; otherwise, it is 0.

[0102] Obtain the boundary decision (e.g., D) for each mode. t D v and D a After that, a multimodal fuzzy decision space is constructed based on the boundary decisions of each modality, specifically including:

[0103] S21. Calculate the boundary ambiguity for each mode;

[0104] S22. Obtain the decision space after fuzzing each mode based on the boundary fuzziness of each mode;

[0105] S23. Construct a multimodal fuzzy decision space based on the decision space after each modality is fuzzy.

[0106] The transition boundaries of different modalities exhibit positional offsets from the scene boundaries, with varying offset lengths. This limits the decision space, hindering subsequent decision fusion and impacting decision accuracy. Therefore, we first calculate the boundary ambiguity for each modality. Ambiguity measures the degree of offset between the transition boundary of each individual modality and the scene boundary, and also represents the membership degree of a shot within the boundary set in that modality. By blurring the boundary decisions for each modality based on the boundary ambiguity of different modalities, we can account for the deviations of different modalities and improve detection accuracy.

[0107] First, the distribution of boundary position offsets for different modes is statistically analyzed in the data set. Then, the boundary ambiguity for different modes is obtained based on this distribution. Finally, the fuzzy decision space is constructed using this boundary ambiguity and the boundary decision for a single mode.

[0108] During implementation, the boundary ambiguity of each mode is calculated using the following procedure:

[0109] S211. Construct a scene boundary dataset; each sample in the scene boundary dataset includes a shot of each video, as well as a decision on whether each shot is a unimodal semantic boundary in each modality and a label on whether it is a scene boundary;

[0110] S212. Calculate the total number of scene boundaries with different offsets for each sample in the scene boundary dataset under each modality.

[0111] S213. Calculate the boundary ambiguity of each mode based on the total number of scene boundaries with different offsets for each sample in the scene boundary dataset under each mode.

[0112] The decision of whether a shot is a unimodal semantic boundary in each modality, that is, whether a shot is an audio sentence / comma boundary in the audio modality, a key person transition boundary in the video modality, and a subtitle transition boundary in the text modality.

[0113] During implementation, the scene boundary dataset constructed is the Scene-NSG-M dataset.

[0114] During implementation, the following formula is used to calculate the total number of scene boundaries with different offsets for each sample in the scene boundary dataset under each modality:

[0115]

[0116] Among them, D m (i) represents the decision result of the i-th shot in the boundary decision of the sample in modality m, d represents the offset length, G(i+d) represents the label of whether the boundary frame of the (i+d)-th shot is the scene boundary, and k represents the boundary decision D of the sample in modality m. m The length of N m (d) represents the total number of scene boundaries with an offset length of d from the scene boundary in the boundary decision of the sample in mode m.

[0117] Where m∈{v,a,t}. When m is v, it represents the video modality; when m is a, it represents the audio modality; and when m is t, it represents the text modality.

[0118] The boundary ambiguity for each mode is calculated using the following formula, based on the total number of scene boundaries at different offsets for each sample in the scene boundary dataset for each mode:

[0119]

[0120] in, N represents the total number of scene boundaries with an offset length of d for the j-th sample in modality m, and N represents the total number of samples. Let P represent the total number of boundary scenes with an offset length of l for the j-th sample in modality m, where K represents the maximum offset length. m (d) represents the boundary ambiguity when the offset length is d in mode m.

[0121] In practice, the maximum offset length K is taken as the number of shots of the sample with the most shots in the dataset.

[0122] After obtaining the boundary ambiguity of each mode, the decision space after ambiguity for each mode is obtained based on the boundary ambiguity of each mode, specifically including:

[0123] For the transition boundary in the boundary decision of each mode, the formula is used. Calculate the fuzzy values ​​to obtain the fuzzy decision space;

[0124] Among them, P m (d) represents the fuzziness of the boundary decision in mode m when the offset length is d, where d represents the offset length, D m (id) represents the decision result of the id-th shot in the boundary decision under modality m. Let k represent the decision space after blurring the decision result of the i-th shot in the boundary decision under modality m, and let k represent the boundary decision D under modality m. m The length.

[0125] In implementation, for each modality's boundary decision, the transition boundary—that is, each non-zero element in the boundary decision—is replaced with the element specified by the formula. The fuzzy values ​​are obtained, thus yielding the fuzzy decision space.

[0126] Obtain the decision space of each modality of the video to be detected after blurring. Subsequently, scene boundary decisions are made based on the multimodal fuzzy decision space to obtain the scene boundary of the video to be detected. If the decision result determines that a certain shot is a scene boundary, then the boundary frame of that shot is the scene boundary frame.

[0127] During implementation, a pre-trained random forest model is used to determine the scene boundaries of the video based on the multimodal fuzzy decision space, and the final scene boundary detection results are obtained.

[0128] In practice, the trained random forest model uses the Classification and Regression Tree (CART) algorithm tree as the decision tree to construct the random forest.

[0129] The random forest model is used to determine whether each shot is a scene boundary. In practice, the fuzzy decision space of each sample in the Scene-NSG-M scene boundary dataset is extracted. And the true label G, construct the training sample set for the random forest. A training sample consists of a shot, the corresponding... The dataset consists of the true label G(i). Since there are far fewer shots belonging to the scene boundary than shots not belonging to the scene boundary in a video, there is a data imbalance, which makes the random forest training process more biased towards negative class decisions. Therefore, this invention oversamples positive samples when training the random forest model to balance the distribution of sample points in the dataset.

[0130] For each positive training sample in a video Determine the s nearest positive samples as the sample x. i The nearest neighbor, randomly select a sample from the nearest neighbor. Then based on the training sample x i and the nearest neighbor sample x j This generates multiple new positive samples, that is, in x i With x j Randomly inserted value points are used as new positive sample points. In practice, the new positive sample point x is calculated using the following formula. new :

[0131] x new =x i +rand(0,1)×(x i -x j )

[0132] Here, rand(0,1) represents generating a random number between 0 and 1. In implementation, it depends on x. i and x j The number of new positive samples generated can be set according to the ratio of positive to negative samples. For example, if the ratio of positive to negative samples in the dataset is 1:30, then 30 new positive samples will be generated for each positive sample, thereby balancing the distribution of sample points in the dataset.

[0133] A trained random forest model is obtained by training a random forest training sample set.

[0134] By inputting the fuzzy decision space of each modality of the video to be detected into a random forest for scene boundary determination, the scene boundary detection results of the video to be detected can be obtained quickly and accurately.

[0135] The multimodal decision fusion video scene boundary detection method of the present invention is not only adaptable to news and variety shows, but also has good generalization performance in sports events with multiple transitions and multiple shot sizes (such as Sanda, boxing and other sports events). It can be seen that the effect of the present invention does not depend solely on the dataset, but is based on the overall video scene boundary detection method to achieve beneficial effects.

[0136] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0137] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A video scene boundary detection method based on multimodal decision fusion, characterized in that, Includes the following steps: Perform shot boundary detection on the video to be inspected to obtain each shot in the video to be inspected; In each modality, determine whether each shot is a transition boundary to obtain the boundary decision for each modality; A multimodal fuzzy decision space is constructed based on the boundary decisions of each modality, and the scene boundary of the video to be detected is obtained by performing scene boundary decisions based on the multimodal fuzzy decision space. A multimodal fuzzy decision space is constructed based on the boundary decisions of each modality, including: Calculate the boundary ambiguity for each mode; The decision space after fuzzing each mode is obtained based on the boundary fuzziness of each mode; A multimodal fuzzy decision space is constructed based on the decision space after each modality is fuzzified; The boundary ambiguity for each mode is calculated using the following procedure: Construct a scene boundary dataset; each sample in the scene boundary dataset includes a shot of each video, as well as a boundary decision of whether each shot is a unimodal semantic boundary in each modality and a label of whether it is a scene boundary; Calculate the total number of scene boundaries with different offsets for each sample in the scene boundary dataset in each modality; The boundary ambiguity of each mode is calculated based on the total number of scene boundaries with different offsets for each sample in the scene boundary dataset for each mode. The following formula is used to calculate the total number of scene boundaries with different offsets for each sample in the scene boundary dataset in each modality: in, Let represent the decision result of the i-th shot in the boundary decision of the sample under mode m, and d represent the offset length. The label indicates whether the (i+d)th shot is a scene boundary, and k represents the boundary decision of the sample in modality m. Length, This represents the total number of scene boundaries with an offset length of d from the scene boundary in the boundary decision of the sample in mode m.

2. The video scene boundary detection method based on multimodal decision fusion according to claim 1, characterized in that, The boundary ambiguity for each mode is calculated using the following formula, based on the total number of scene boundaries at different offsets for each sample in the scene boundary dataset for each mode: in, N represents the total number of boundary scenes with an offset length of d for the j-th sample in mode m, and N represents the total number of samples. This indicates that the offset length of the j-th sample in mode m is... l The total number of boundary scenes, where K represents the maximum offset length. This represents the boundary ambiguity when the offset length is d under mode m.

3. The video scene boundary detection method based on multimodal decision fusion according to claim 1, characterized in that, The decision space after fuzzing each mode is obtained based on the boundary fuzziness of each mode, including: For the transition boundary in the boundary decision of each mode, the formula is used. Calculate the fuzzy values ​​to obtain the fuzzy decision space; in, This represents the fuzziness of the boundary decision in mode m when the offset length is d, where d represents the offset length. Let represent the decision result of the id-th shot in the boundary decision under mode m. Let k represent the decision space after blurring the decision result of the i-th shot in the boundary decision under modality m, and k represent the boundary decision under modality m. The length.

4. The video scene boundary detection method based on multimodal decision fusion according to claim 1, characterized in that, The multimodal approach includes a text modality. In the text modality, determining whether each shot represents a transition boundary yields boundary decisions for the text modality, including: The title sequence is obtained by detecting the title in the keyframes of each shot; Based on the similarity of adjacent thematic subtitles in the thematic subtitle sequence, we determine whether each shot is a transition boundary, thus obtaining the boundary decision of the text modality.

5. The video scene boundary detection method based on multimodal decision fusion according to claim 4, characterized in that, The similarity between two adjacent title tags is calculated using the following formula: ； If similarity If the value is less than the first threshold, then the shot corresponding to the i-th theme subtitle is the transition boundary; in, This represents the i-th theme subtitle in the theme subtitle sequence. This represents the (i+1)th theme subtitle in the theme subtitle sequence. express and The intersection, express and The union of, A function that calculates the number of elements.

6. The video scene boundary detection method based on multimodal decision fusion according to claim 1, characterized in that, The multimodal approach includes a video modality. In the video modality, determining whether each shot represents a transition boundary yields boundary decisions for the video modality, including: Detect facial features in keyframes of each shot; If the similarity between the detected facial feature and the facial feature of the key person exceeds the second threshold, then the shot corresponding to the detected facial feature is the conversion boundary; thus, the boundary decision of the video modality is obtained.

7. The video scene boundary detection method based on multimodal decision fusion according to claim 1, characterized in that, The multimodal nature includes an audio modality. Determining whether each shot represents a transition boundary within the audio modality yields boundary decisions for the audio modality, including: The audio signal of the video to be detected is extracted, and the silent region in the audio signal is obtained based on the root mean square energy. The shot corresponding to the start time of the silent zone is the transition boundary; the boundary decision of the audio modality is obtained.

Images