Lens boundary detection method and device based on double-branch parallelism and storage medium

Abstract

The invention discloses a shot boundary detection method and device based on double-branch parallelism and a storage medium, and the method comprises the steps: firstly, for an input video frame, extracting multi-scale time sequence features through employing the stacking of a plurality of cavity deep convolutional neural networks; then, through double-branch parallelism, respectively using the color histogram features and the similarity matrix, and combining the time sequence features to predict the probability that the video frame is in an instantaneous switching state and a gradual switching state; and finally, obtaining a final lens boundary detection result through optimization processing. The video gradient switching detection method is used for solving the problem of gradient switching detection in a video, designs a double-branch parallel detection mode, captures sudden instantaneous switching through color histogram features, captures a continuously accumulated gradient switching process through a similarity matrix, optimizes a gradient switching detection result through a repair strategy, and improves the detection accuracy. And the detection accuracy of the boundaries of the two lenses is improved. The method can provide powerful support for applications such as intelligent video segmentation and video editing.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision and relates to temporal boundary detection. It is a shot boundary detection method, device and storage medium based on dual-branch parallelism. Background Art

[0002] With the widespread adoption of the internet and the rapid development of multimedia technology, the generation and dissemination of video data has reached unprecedented scale, leading to a growing demand for efficient and intelligent analysis and processing of massive amounts of video data. Shot boundary detection is a fundamental task in video understanding, aiming to quickly segment long videos into independent shot segments. Generally speaking, shot cuts can be categorized into two types: instantaneous cuts and gradual cuts. Instantaneous cuts occur when the first frame of the subsequent shot immediately follows the last frame of the previous shot, creating an instantaneous effect. In contrast, gradual cuts involve adding a transition between two shots during post-editing. With the advancement of video editing and editing technology, gradual cuts are increasingly being used in videos to enhance the coherence between shots and improve the viewing experience. These cuts are now being used in a variety of forms. However, existing shot boundary detection techniques still have shortcomings in detecting gradual cuts, resulting in inaccurate and incomplete detection results. Specifically, some existing methods do not distinguish between gradual switching and instantaneous switching, but also use a single frame to represent an instantaneous switching. This approach ignores the continuity of gradual switching and cannot indicate its start and end time. Another part of the method uses multiple frames to represent the detection results of gradual switching, but does not take into account the different characteristics of the boundaries between the two types of shots. Instead, they use the same scheme to detect them uniformly. For example, they all use the similarity between adjacent video frames as the recognition basis, and use the instantaneous jump and gradual curve change patterns of similarity to detect shot boundaries. However, in gradual switching, the similarity between adjacent frames cannot well reflect the significant characteristics of the gradual process, which can easily lead to inaccurate final gradual switching detection results and omissions. Summary of the Invention

[0003] The technical problem to be solved by the present invention is that existing shot boundary detection technology is difficult to accurately detect gradual shot boundaries in videos, especially when there are complex and diverse gradual switching in the video, which leads to the following problems:

[0004] 1) Inaccurate detection results: Existing technologies can only detect one or several frames in a series of gradual transitions.

[0005] 2) Detection omissions: Sharp mutations rarely occur in gradual transitions. As a result, existing shot boundary detection methods, such as adjacent frame similarity detection, may mistakenly identify some gentle gradual transitions as part of a shot, failing to detect the shot boundary.

[0006] 3) Inconsistent detection: There are breaks in the multi-frame results of the gradual switching detection, and it is impossible to form a complete detection result.

[0007] In response to the above problems, the present invention aims to improve the accuracy and comprehensiveness of detecting gradual switching in shot boundaries through a shot boundary method based on dual-branch parallelism.

[0008] The technical solution of the present invention is: a shot boundary detection method based on a dual-branch parallel process, which adopts a dual-branch parallel structure to simultaneously predict the probability of instantaneous switching and gradual switching of the video frame for the input video, including the following steps:

[0009] 2) Multi-scale temporal feature extraction:

[0010] For the input video, a sliding window mechanism is used to construct a video frame sequence. Multiple hole deep convolutional neural network units are stacked to improve the receptive field, extract features from different time scales, and perform multi-scale temporal feature extraction on the video frame sequence.

[0011] 2) Dual-branch parallel prediction:

[0012] The instantaneous switching detection branch and the gradual switching detection branch share the multi-scale temporal features extracted by the dilated deep convolutional neural network, and each uses different detection schemes based on the characteristics of instantaneous switching and gradual switching:

[0013] 2.1) In the instantaneous switching detection branch, color histogram features are directly extracted from the input video frame and concatenated with multi-scale temporal features as the input to the instantaneous switching classifier;

[0014] 2.2) In the gradual transition detection branch, based on the multi-scale temporal features, the similarity between all video frames within the sliding window is calculated to obtain a similarity matrix. The similarity results corresponding to the video frames are then concatenated with the multi-scale temporal features and used as the input to the gradual transition classifier.

[0015] 2.3) Based on their respective inputs, the two branches predict the probability of each frame within the window being an instantaneous switch or a gradual switch;

[0016] 3) Shot boundary acquisition:

[0017] Based on the prediction results of the instantaneous switching detection branch and the gradual switching detection branch for each frame, the positions of the two shot boundaries in the video are obtained in combination with a set threshold, and the final shot boundary detection result is obtained. The threshold is set according to the prediction precision of the boundary frame.

[0018] Furthermore, the prediction results of the gradient switching detection branch are patched and processed as follows:

[0019] 3.1) Based on the predicted probability results of each frame and the set threshold, determine whether each frame is in a gradual transition and obtain all the gradual transition intervals in the video frame sequence;

[0020] 3.2) For all the initially predicted gradient switching intervals, if the distance between two intervals is less than the interval length, these two intervals are merged into a whole interval. This process is repeated until there are no intervals that need to be merged, thereby maintaining the integrity of the gradient switching detection.

[0021] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program segment, and the at least one instruction or at least one program segment is loaded and executed by the processor to implement the above-mentioned dual-branch parallel shot boundary detection method, thereby simultaneously predicting shot boundaries for instantaneous switching and gradual switching of an input video.

[0022] The present invention also provides a storage medium storing a computer executable program. When the program is executed by a processor, the above-mentioned shot boundary detection method based on dual-branch parallelism is implemented.

[0023] The present invention proposes a solution that can accurately detect instantaneous switching and gradual switching at the same time. For multiple shot switching in a video, there is no need for cyclic iterative detection. The present invention adopts a dual-branch parallel method. According to the suddenness of instantaneous switching and the continuity of gradual switching, different solutions are adopted in the corresponding branches respectively, and the detection is combined with the temporal characteristics of the video frame. It can better cope with the continuous shot boundary detection of long video segments. Furthermore, the prediction results of gradual switching are patched and processed to further optimize the detection results of gradual switching. Compared with the existing technology that uses a single feature as the basis for unified detection, the present invention effectively improves the accuracy of gradual switching detection based on the dual-branch parallel shot boundary detection method.

[0024] The present invention proposes a shot boundary detection method based on dual-branch parallelism. Based on the different characteristics of instantaneous switching and gradual switching in shot boundaries, branch design is performed separately. Combined with multi-scale temporal features, this method can provide the classifier with richer and more comprehensive semantic information, ultimately improving the accuracy of gradual switching detection. Compared with the existing technology, the present invention has the following advantages:

[0025] 1) Improving the accuracy of shot boundary detection: By designing a separate detection branch for gradual transitions, the present invention fully utilizes the persistence of gradual transition detection. While ensuring the effectiveness of instantaneous transition detection, it also significantly improves the accuracy of gradual transition detection results. In particular, for gradual transitions with long durations, the detection results of the present invention can overlap with the actual boundary for multiple frames.

[0026] 2) Improved comprehensiveness of shot boundary detection: Through a dual-branch parallel structure, each branch can focus on processing different types of shot boundaries, more closely matching the characteristics of instantaneous and gradual switching shots. This improves the comprehensiveness of the detection results for both instantaneous and gradual switching. For input videos containing multiple shot boundaries, the present invention can more comprehensively cover all types of shot boundaries in the video.

[0027] 3) Enhancing the consistency of gradual cut detection results: By performing post-processing on the prediction results of the gradual cut detection branch and repairing minor breaks in the initial prediction results, the present invention significantly improves the consistency of the gradual cut detection results. The repaired detection results are more complete and more closely match the actual gradual cut shot boundaries, thereby improving the ultimate accuracy of the gradual cut detection results. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is the overall framework diagram of the method of the present invention.

[0029] Figure 2 It is a schematic diagram of the post-repair processing in the present invention.

[0030] Figure 3 This is a display of the shot boundary detection results for instantaneous switching under different test samples of the present invention.

[0031] Figure 4 This is a display of the shot boundary detection results for gradual switching under different test samples of the present invention. DETAILED DESCRIPTION

[0032] This paper proposes a dual-branch parallel shot boundary detection method. By branching the detection of instantaneous and gradual transitions, and targeting the different characteristics of the two shot boundaries, a branching scheme and optimization strategy are designed to accurately detect multiple shot boundaries in complex videos. This method not only significantly improves the accuracy, comprehensiveness, and consistency of detection of complex, continuous, gradual transitions, but also helps further improve the accuracy of instantaneous transition detection, promoting the application of shot boundary detection technology in scenarios such as intelligent video segmentation, video editing, and positioning, and providing a solid foundation for video understanding.

[0033] The present invention realizes accurate detection of shot boundaries through a dual-branch parallel structure. Figure 1 As shown, the present invention includes three key modules: a multi-scale temporal feature extraction module, an instantaneous switching detection branch, and a gradual switching detection branch. Based on these three modules, temporal feature extraction, instantaneous switching detection, and gradual switching detection are realized. Based on the input video frame sequence, the instantaneous and gradual switching shot boundaries in the video are detected.

[0034] The specific implementation process of the present invention is as follows:

[0035] 1) Extracting multi-scale temporal features based on a dilated deep convolutional neural network:

[0036] In the temporal feature extraction stage, the present invention utilizes multiple dilated deep convolutional neural networks as the core tool for extracting multi-scale temporal features of videos, and progressively extracts multi-scale temporal features from video frame sequences layer by layer. The temporal feature extraction part of the present invention combines dilated convolution and deep convolutional neural network (DCNN). The design of dilated convolution can effectively expand the receptive field of the convolution kernel, thereby capturing long-term dependencies and fine-grained temporal changes in the video, which provides sufficient and rich temporal information for the subsequent shot boundary detection stage. Specifically, each dilated deep convolutional neural network unit DDCNN can extract features from different time scales at the same time by applying dilated convolution with four different dilation rates of 1, 2, 4, and 8 to the same input. The smaller dilation rate focuses on subtle changes in a short period of time, and the larger dilation rate captures the overall trend over a long time span, taking into account both rapid mutations and continuous gradual changes. The dilated deep convolutional neural network units are grouped into groups of two, with skip connections within the group. The output of the previous unit serves as the input of the next unit, which is then added to the output of the next unit. Spatial average pooling is then performed to serve as the input of the next group. Skip connections ensure that fine-grained information from lower layers can be passed to subsequent layers, improving feature fusion. Spatial average pooling reduces computational effort, allowing subsequent dilated convolutions to extract temporal features based on a more compact feature representation, further aggregating and abstracting spatial information. The dilated deep convolutional neural network is stacked sequentially, and as the number of layers increases, the network's overall receptive field expands. Ultimately, the last unit can cover a time range of up to 97 frames, thereby capturing long-term dependencies.

[0037] 2) Instantaneous switching and gradual switching prediction based on dual-branch parallelism

[0038] The present invention uses independent branches to distinguish and detect instantaneous switching and gradual switching of lenses, specifically:

[0039] 2.1) In the instantaneous cut detection branch, color histogram features are directly extracted from the input video frames and concatenated with multi-scale temporal features to serve as the input for the instantaneous cut classifier. Given the sudden nature of instantaneous cuts, noticeable and dramatic visual changes occur between the two frames at the cut location. Color histogram features extracted directly from the input video frames are used to capture these changes.

[0040] 2.2) In the gradual handoff detection branch, the similarity between all video frame pairs within the sliding window is calculated based on multi-scale temporal features to obtain a similarity matrix. Here, a video frame pair refers to any combination of two video frames within the sliding window, regardless of whether they are adjacent. The similarity result is then concatenated with the multi-scale temporal features and used as the input for the gradual handoff classifier. Given the continuous nature of gradual handoffs, there are no drastic changes between adjacent frames. Instead, changes gradually accumulate over multiple frames. Using the similarity matrix, the similarity of multiple video frame pairs within the sliding window is calculated based on the multi-scale temporal features to capture these continuous changes.

[0041] This paper innovatively proposes a dual-branch parallel structure that distinguishes between instantaneous and gradual switching. For each frame in an input video sequence, the probability of it being in an instantaneous or gradual switching state is predicted. To address the sudden nature of instantaneous switching and the persistence of gradual switching, the parallel dual-branch architecture uses color histograms and similarity matrices, respectively, to effectively detect sudden, dramatic changes and cumulative, persistent changes in the video.

[0042] Specifically, instantaneous cuts are often accompanied by dramatic changes in visual features. Statistical features fluctuate dramatically within a very short period of time, creating a distinct discontinuity in visual perception. Video sequences inherently have a high degree of redundancy, and closely spaced frames often exhibit high similarity. In particular, consecutive frames within the same shot often exhibit high consistency in composition, lighting, and color. Therefore, once a shot cut occurs, the differences between adjacent frames increase significantly, forming a distinct boundary. In the instantaneous cut detection branch, color histogram features are directly extracted from the input video frame sequence. These features effectively reflect sudden and significant changes in the video. These features are then combined with the multi-scale temporal features extracted by a dilated deep convolutional neural network and serve as the input to the instantaneous cut classifier. Through this feature fusion approach, the instantaneous cut classifier can leverage the sudden change signals revealed by the color histogram and the contextual information provided by the temporal features to predict the probability of an instantaneous cut for each frame in the video sequence.

[0043] Gradual switching is a continuous and coherent process, manifested as a smooth transition between two shots. It typically does not exhibit drastic changes in visual features between adjacent video frames, but rather accumulates over time, presenting a gradual change. Furthermore, within the same shot, frame changes are often affected by factors such as camera motion, background changes, or subject motion. Therefore, in the gradual switching detection branch, a similarity matrix is calculated based on multi-scale temporal features extracted by a dilated deep convolutional neural network to measure the similarity between video frame pairs within a sliding window, thereby better capturing subtle changes between frames. This similarity matrix not only effectively represents the relationship between adjacent frames but also summarizes the similarity relationships between video frame pairs at different distances within the window, reflecting the pattern of change. The similarity calculation structure is combined with the multi-scale temporal features and used as the input to the gradual switching classifier. This design allows the gradual switching classifier to comprehensively consider the relationships between multiple frames, identify specific change patterns, and effectively predict the probability of each frame in the video sequence being in the gradual switching process.

[0044] 3) Shot boundary acquisition

[0045] Shot boundary detection uses the dual-branch frame-by-frame shot boundary probability prediction results, combined with a set threshold, to obtain instantaneous cut boundary detection results and gradual cut prediction intervals. In this embodiment, the preferred threshold is set to 0.5. Given that the predicted probability results are between 0 and 1, 0.5 is a commonly used threshold for binary classification tasks. A higher threshold predicts fewer boundary frames, while a lower threshold predicts more boundary frames.

[0046] In the prior art, the adjacent frame distance detection method only considers the differences between adjacent frames, while the differences in gradual switching are continuously accumulated across multiple frames. For a segment of length L, compared to the adjacent frame distance sequence of size L-1, the similarity matrix of size L*L used in the present invention contains richer similarity information between video frame pairs, and is therefore more conducive to the detection of gradual switching. At the same time, unlike the prior art that only considers the changes in the image features of the video frame itself, the present invention not only utilizes the video image features, but also considers the continuity of the video, extracting multi-scale temporal features and splicing the features of the two branches as the input of the classifier. Especially for gradual switching, the multi-scale temporal features represent the characteristics of each video frame at different temporal scales, and the similarity matrix reflects the overall change pattern of the video within the sliding window. Compared with the similarity matrix calculated based on low-level visual features, the similarity matrix calculated based on multi-scale temporal features can more accurately and meticulously represent temporal changes. Combined with the multi-scale temporal features, it can provide the classifier with richer and more comprehensive semantic information, thereby improving the final detection accuracy of gradual switching.

[0047] Since the predicted probability of each frame in the video sequence being in the gradient switching process is different, and the change speed at different moments within the gradient switching is not necessarily uniform, it is possible that individual frames may have low scores, resulting in incoherent gradient switching detection. Specifically, for a continuous gradient switching, the initial prediction results may be some sub-segments, and these intervals are very close. Based on this, the present invention further performs post-processing on the prediction results of the gradient switching detection branch, and the post-processing merges these intervals, such as Figure 2 As shown, for each gradient transition segment naturally delineated by the initial detection results, the distance between it and the adjacent segment is calculated. If the distance between them is less than the length of the two segments, the two segments should be merged. The detection of segments that need to be merged is performed in a loop. If a segment merge occurs, a new round of merge detection is required. This continues until the last round of no merges occurs, at which point the final gradient transition detection result is obtained. This post-processing can effectively detect and repair minor breaks in the initial detection results, making the final gradient transition detection result closer to the true boundary.

[0048] The present invention preferably trains its model on a high-quality shot boundary detection dataset to ensure that the model can more accurately capture the characteristics of instantaneous switches and various complex gradual switches. To ensure the high quality, comprehensiveness, and challenging nature of the training data, the present invention constructs a shot boundary detection dataset. The shot cuts in sports scene videos are rich and have a higher switching frequency than existing datasets, making it suitable for shot boundary detection tasks. This embodiment uses a game video in a sports scene. The video is clear and stable, with an image quality of no less than 720p. The video contains rich and high-frequency shot cut boundaries, and the sub-segments divided from it are detailedly annotated with shot boundaries, as follows.

[0049] 1) Data collection and preprocessing: Filter high-resolution sports video clips, segment them into sub-segments with a high number of shot boundaries, and unify the frame rates of all video segments;

[0050] 2) Instantaneous switching annotation: For all instantaneous switching in the video clip, locate the exact location where it occurs and mark the two frames before and after the instantaneous switching occurs;

[0051] 3) Gradual transition annotation: For all the gradual transitions in the video clip, locate their start and end positions respectively, and mark the first and last frames of the gradual transition.

[0052] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program segment, and the at least one instruction or at least one program segment is loaded and executed by the processor to implement the above-mentioned dual-branch parallel shot boundary detection method, thereby simultaneously predicting shot boundaries for instantaneous switching and gradual switching of an input video.

[0053] The present invention also provides a storage medium storing a computer executable program. When the program is executed by a processor, the above-mentioned shot boundary detection method based on dual-branch parallelism is implemented.

[0054] The present invention is implemented based on a computer program. Based on this, the present invention also provides an electronic device comprising a processor and a memory. The memory stores at least one instruction or at least one program segment. The at least one instruction or at least one program segment is loaded and executed by the processor to implement the aforementioned dual-branch parallel shot boundary detection method, simultaneously predicting shot boundaries for both instantaneous and gradual switching of an input video. Also provided is a computer-readable storage medium storing at least one instruction or at least one program segment. When executed, the at least one instruction or program segment implements the aforementioned dual-branch parallel shot boundary detection method. The instructions or program code for implementing the method of the present invention can be written in any combination of one or more programming languages. The instructions or program code can be executed entirely on the data processor, partially on the processor, as a standalone software package, partially on the processor and partially on a remote device, or entirely on a remote device or server.

[0055] like Figure 3 and Figure 4 The figures show the detection performance of the present invention for instantaneous and gradual transitions in different test cases. The results show that the shot boundaries detected by the present invention closely coincide with the actual shot boundaries. The detection results for instantaneous transitions are completely consistent with the actual boundaries, and the detection results for gradual transitions of varying durations are essentially consistent with the actual boundaries, showing good consistency.

Claims

1. A shot boundary detection method based on dual-branch parallelism, characterized by A dual-branch parallel structure is used to simultaneously predict the probability of instantaneous switching and gradual switching of the input video frame, including the following steps: 1) Multi-scale temporal feature extraction: For the input video, a sliding window mechanism is used to construct a video frame sequence. Multiple hole deep convolutional neural network units are stacked to improve the receptive field, extract features from different time scales, and perform multi-scale temporal feature extraction on the video frame sequence. 2) Dual-branch parallel prediction: The instantaneous switching detection branch and the gradual switching detection branch share the multi-scale temporal features extracted by the dilated deep convolutional neural network, and each uses different detection schemes based on the characteristics of instantaneous switching and gradual switching: 2.1) In the instantaneous switching detection branch, color histogram features are directly extracted from the input video frame and concatenated with multi-scale temporal features as the input to the instantaneous switching classifier; 2.2) In the gradual transition detection branch, based on the multi-scale temporal features, the similarity between all video frames within the sliding window is calculated to obtain a similarity matrix. The similarity results corresponding to the video frames are then concatenated with the multi-scale temporal features and used as the input to the gradual transition classifier. 2.3) Based on their respective inputs, the two branches predict the probability of each frame within the window being an instantaneous switch or a gradual switch; 3) Shot boundary acquisition: Based on the prediction results of the instantaneous switching detection branch and the gradual switching detection branch for each frame, the positions of the two shot boundaries in the video are obtained in combination with a set threshold, and the final shot boundary detection result is obtained. The threshold is set according to the prediction precision of the boundary frame.

2. The shot boundary detection method based on dual-branch parallelism according to claim 1, characterized in that The prediction results of the gradient switching detection branch are patched and processed as follows: 3.1) Based on the predicted probability results of each frame and the set threshold, determine whether each frame is in a gradual transition and obtain all the gradual transition intervals in the video frame sequence; 3.2) For all the initially predicted gradient switching intervals, if the distance between two intervals is less than the interval length, these two intervals are merged into a whole interval. This process is repeated until there are no intervals that need to be merged, thereby maintaining the integrity of the gradient switching detection.

3. The shot boundary detection method based on dual-branch parallelism according to claim 1, characterized in that Independent branches are used to distinguish and detect instantaneous and gradual shot switches. During training, two cross-entropy losses are used for the prediction results of the two branches and the actual shot boundaries, namely: minimizing the difference between the instantaneous switch prediction results and the actual instantaneous switch, and the difference between the gradual switch prediction results and the actual gradual switch; during inference, the two branches each predict the probability of each input video frame being an instantaneous switch and a gradual switch.

4. The shot boundary detection method based on dual-branch parallelism according to claim 1, characterized in that In multi-scale temporal feature extraction, an even number of atrous deep convolutional neural network units are stacked. Each atrous deep convolutional neural network unit is grouped into two, and a jump connection is used within the group. The output of the previous unit is used as the input of the next unit, and the output of the previous unit is added to the output of the next unit, and then spatial average pooling is performed as the input of the next group.

5. The shot boundary detection method based on dual-branch parallelism according to claim 1, characterized in that In step 3), for the prediction results of each frame, both branches use the same threshold to judge, and the threshold is set to 0.

5.

6. The shot boundary detection method based on dual-branch parallelism according to claim 1, characterized in that A shot boundary detection dataset is constructed for training a dual-branch shot boundary detection model. The shot boundary detection dataset is constructed based on game videos in sports scenes and performs shot boundary annotation, including: 4.1) Data Collection and Preprocessing: We screen high-resolution sports video clips with a quality of at least 720p, segment them into sub-segments containing multiple shot boundaries, and standardize the frame rate of all sub-segments. 4.2) Transient Switch Annotation: For all transient switches in a sub-segment, locate the exact location where they occur and mark the two frames before and after the transient switch occurs; 4.3) Gradient Switch Annotation: For all gradient switches in a sub-segment, locate their start and end positions respectively, and mark the first and last frames of the gradient switch.

7. An electronic device, characterized in that The device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program segment, and the at least one instruction or at least one program segment is loaded and executed by the processor to implement the shot boundary detection method based on dual-branch parallelism according to any one of claims 1 to 6.

8. A storage medium characterized by storing There is a computer executable program, which, when executed by a processor, implements the shot boundary detection method based on dual-branch parallelism as described in any one of claims 1 to 6.

Images