Self-supervised video scene boundary detection method based on time sequence scene creator
Through the self-supervised method of the timing scene creator, high-quality video scene boundary and non-boundary lenses are generated, which solves the problem of pseudo-boundary construction neglecting contextual relationships in the existing technology, and improves the accuracy of video scene boundary detection and the self-model of the model Supervision capability.
Patent Information
- Application Number
- CN202411216320.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-02
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-09-02
AI Technical Summary
The existing video scene segmentation method ignores contextual relationships when constructing pseudo-boundaries, resulting in flaws in label data, affecting the accuracy of video scene boundary detection.
Using a self-supervised method based on the timing scene creator, multiple combined scene segments are generated through the scene boundary generation module and the non-scene boundary generation module. High-quality pseudo-boundary and non-boundary lenses are constructed through sliding window and lens exchange technology, for training Video scene boundary detection model.
The accuracy of video scene boundary detection and the self-supervision of the model are improved, and the generated data quality is higher, which enhances the video scene segmentation capability of the model.
Smart Images

Figure CN119007085A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of pseudo-label generation and video understanding, and specifically is a self-supervised video scene boundary detection method based on a temporal scene creator. Background Art
[0002] Deep learning has made great progress in many fields, but the foundation of this progress is the support of a large amount of data. In the video field, it is particularly difficult to obtain a large amount of labeled data through manual annotation due to the long time span of the video. Artificially constructed labeled data as a supervisory signal to train neural networks has become a feasible way to alleviate the demand for labeled data.
[0003] For the video scene segmentation task, the existing methods for creating pseudo boundaries are directly or indirectly spliced from video clips, and the semantic relationship between the spliced clips is completely determined by the temporal distance of the clips. However, these methods consider a relatively single situation, the contextual relationship is not fully considered, and the non-scene boundary shots are mainly randomly selected, ignoring the situation of the selected scene boundary shots, so there are certain flaws in the label data. This problem prompted us to explore how to better construct pseudo boundaries and non-boundary shots to improve the model's video scene boundary detection capabilities. Summary of the invention
[0004] The purpose of the present invention is to provide a more reasonable and effective pseudo scene boundary construction method, thereby improving the video scene boundary detection capability.
[0005] In order to achieve the above object, the present invention adopts such a technical solution: a self-supervised video scene boundary detection method based on a temporal scene creator, comprising the following steps performed in sequence:
[0006] Step S1: Extraction of shot sequence: Get an input video, get the shots in it, and form a shot sequence S = [s1, s2, ..., s t ], where s1, s2, s t The first shot, the second shot and the t-th shot are respectively used to sample key frames from the shots; the existing method is used to extract a few frames from the shots in each video. Generally, each shot has more than ten to twenty frames. In the experimental part, we extracted three frames as key frames. The method of extracting key frames adopts the existing method.
[0007] Step S2: Construct a temporal boundary creator network model TSC, which includes a parallel scene boundary generation module SBG and a non-scene boundary generation module NSBG; the SBG module is used to generate scene pseudo boundaries and enhanced boundaries; the NSBG module is used to generate non-scene boundaries; the shot group in the sliding window randomly generates a shot sequence as training data through the SBG module or the NSBG module to achieve self-supervised video scene segmentation. The following is the working process of the two modules.
[0008] Step S2-1: Use the visual encoder to obtain i The features are extracted from the corresponding key frames and used as the features of the shot. The features of all shots construct the shot semantic feature sequence X = [x1, x2, ..., x t ], where x1,x2,x t They are lenses s1, s2, s t The lens feature of , t is the length of the sequence of lens and lens semantic features.
[0009] Apply unsupervised time series segmentation to the extracted X to generate M consecutive and non-overlapping video clips, which form a pseudo scene set P = {p i |X=∪ i∈M p i}, where p i is the ith pseudo scene, a pseudo scene p i Corresponds to a video clip.
[0010] Step S2-2: randomly process the video clips through SBG or NSBG to obtain several labeled B A and labeled N.
[0011] In SBG, a set of continuous shot segments B of length k is sampled in the pseudo scene using a sliding window z L , from B L Sampling shot fragment B of length k in the pseudo scene not in R , and B L and B R The pseudo scenes do not intersect, and B L May come from multiple pseudo scenes, B L and B R Spliced together, we get a set of synthetic shot sequences B, B with a length of 2k L and B R The shots at the splicing point are the boundaries of the synthetic video scene, and B is L and B R The shots in the scene are randomly exchanged to obtain the scene boundary shot sequence B after boundary enhancement A ; B L and BR The shot at the splicing point, that is, the middle shot of the shot sequence, is the pseudo-boundary shot, and this shot is labeled as a boundary.
[0012] In NSBG, a sliding window is used to sample N consecutive shots of length k in a pseudo scene. L , in N L Then continue to sample the shot sequence N with a length of k R , splicing N L and N R Get the non-scene boundary shot sequence N. L and N R The shot at the splicing point, that is, the middle shot of the shot sequence, is a non-pseudo-boundary shot, and this shot is marked with a non-boundary label. It should be noted that: in a pseudo scene, a continuous shot N with a length of k is used L Then, we use a shot sequence N of length k. R When there are not enough shots in the pseudo scene, the missing shots are sampled from the next pseudo scene of the pseudo scene.
[0013] Step S3: construct and train a boundary detection model, which includes a sequentially connected context encoder and a scene boundary classifier, a number of labeled B A and labeled N constitute a training set, one of which B A and N are each a training sample.
[0014] Step S3-1: pre-training the boundary detection model using the training samples in the training set to obtain a suboptimal boundary detection model;
[0015] The training samples are input into the boundary detection model, and the output is the predicted label of each training sample. The training loss L is calculated using binary cross entropy. p , as shown in the following formula:
[0016]
[0017] where f s (·) is the scene boundary classifier, It is B A The corresponding training samples after being encoded by the context encoder are is the training sample corresponding to N after being encoded by the context encoder.
[0018] The parameters of the boundary detection model are continuously updated according to the training loss until the maximum number of training times is reached or the loss stops decreasing, and the training is completed to obtain a suboptimal boundary detection model.
[0019] Step S3-2: Use the training samples in the training set to fine-tune the boundary detection model to obtain a suboptimal boundary detection model. Use the training samples to fine-tune the trained TSC, and use the following formula to calculate the fine-tuning loss L in the fine-tuning stage f :
[0020]
[0021] where f b (·) is the scene boundary classifier in the suboptimal boundary detection model, which needs to be trained from scratch. i is the true label of the training sample. The parameters of the suboptimal boundary detection model are continuously updated according to the loss until the maximum number of fine-tuning times is reached or the fine-tuning loss no longer decreases, and the fine-tuning is completed to obtain the optimal boundary detection model.
[0022] Step S4: Obtain the shot sequence S' of the segment to be detected through step S1, input S' into step S21 to obtain the feature sequence X', input X' into the optimal boundary detection model, and output a label indicating whether the middle shot of the video segment to be detected is a scene boundary.
[0023] Further, in step S2-2, obtain B A The steps include:
[0024] Note B L ={s l-k+1 ,...,s l},s l-k+1 is the first shot of the sliding window, s l is the last shot in the sliding window, B R ={s r ,...,s r+k-1},s r For B L A random shot in a pseudo-scene where s r+k-1 For r In the kth shot after that, B L and B R Splicing to get B = {s l-k+1 ,...,s l ,s r ,...,s r+k-1}, it should be noted that B L With B R The pseudo scenes where the shots in B are located do not intersect, and L May come from multiple pseudo-scenarios, i.e. And P R =PP L .s r is a pseudo boundary shot. The scene boundary generation is expressed as follows:
[0025]
[0026] Where b(·) extracts the boundary of the synthetic scene, and B L ={s l-k+1 ,...,s l} and B R ={s r ,...,s r+k-1 The shots in} are divided into the following order according to the number α and probability β e Exchange and get the final scene boundary shot fragment B A :
[0027]
[0028] B A The left half of The right half is represented by Then the scene boundary enhancement can be expressed as follows:
[0029]
[0030] where b(·) extracts the boundary of the synthetic scene, s r is the final pseudo scene boundary.
[0031] Furthermore, in step S2-2, the step of obtaining the non-scene boundary shot sequence N includes:
[0032] Remember N L ={s i-k+1 ,...,s i},s i-k+1 is the first shot of the sliding window, s i is the last shot of the sliding window, when s i N L At the end of the pseudo scene, N R From i Start sampling and get N R ={s i ,...,s i+k-1}; When s i Not for N L At the end of the pseudo scene, N R Then, we can get N R ={s i+1 ,...,s i+k}; N L and N R The final non-scene boundary shot fragment N is obtained by splicing, where s i+1 For i The next shot, s i+k Fori After the k-th shot, N R It can be expressed as follows:
[0033]
[0034] Non-scene boundary generation can be expressed as follows:
[0035]
[0036] in Extract non-boundaries of synthetic scenes.
[0037] Compared with the prior art, the present invention has at least the following beneficial effects:
[0038] The present invention generates multiple sets of synthetic scene segments through TSC. In the SBG of the TSC model, pseudo boundaries are synthesized by splicing video segments, and boundary enhancement is achieved by exchanging a certain number of shots; in the NSBG module, the most likely non-boundary scene is provided by adjacent shots from the same pseudo scene or shots at the end of repeated pseudo scenes. Through the above two modules, multiple synthetic scene sequences can be generated, providing higher quality data for many video scene segmentation models, realizing model self-supervision, and thus improving the video scene boundary detection capability. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] Figure 1 It is a diagram of the structure of the TSC network of the present invention.
[0040] Figure 2 This is the overall flow chart of using the TSC network to generate scene fragments to provide training data for other scene segmentation models.
[0041] Figure 3 It is a detailed schematic diagram of scene generation and non-scene generation of the present invention.
[0042] Figure 4 It is a schematic diagram of scene boundary enhancement of the present invention.
[0043] Figure 5 It is a schematic diagram of a special case of non-scene generation of the present invention. DETAILED DESCRIPTION
[0044] In order to make the purpose, technical scheme and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only partial embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0045] The method of the present invention uses a visual encoder pre-trained on a large-scale image dataset to extract shot-level features, for example, ResNet50 pre-trained on ImageNet is used to extract features.
[0046] Example 1: Freeze the parameters of the visual encoder during all training processes and do not participate in training. The Adam optimizer is used in pre-training, and the lens classifier is MLP; the SGD optimizer is used for fine-tuning, and the lens classifier is also MLP, but the pre-trained model weights are not loaded, that is, this lens classifier needs to be trained from scratch. The learning rate is set to 1e-5 during both pre-training and fine-tuning, and the training is performed for 11 cycles. The first cycle is used for warm-up, and the Cosine scheduler is used to reduce the learning rate in subsequent cycles. In the process of generating pseudo-label data, the lens feature exchange probability β e is 0.2, the upper limit of the lens exchange logarithm U is 4, and the number of shots in each pseudo scene is shot p is 14.
[0047] A self-supervised video scene boundary detection method based on a temporal scene creator comprises the following steps performed in sequence:
[0048] Step S1: Extraction of shot sequence: Extraction of shot sequence: Get an input video, get the shots therein, and form a shot sequence S = [s1, s2, ..., s t ], where s1, s2, s t The first shot, the second shot and the t-th shot are respectively used to sample key frames from the shots; the existing method is used to extract a few frames from the shots in each video. Generally, each shot has more than ten to twenty frames. In the experimental part, we extracted three frames as key frames. The method of extracting key frames adopts the existing method.
[0049] Step S2: Construct a timing boundary creator network model TSC, such as Figure 1 As shown in FIG, TSC includes a scene boundary generation module SBG and a non-scene boundary generation module NSBG in parallel; the SBG module is used to generate scene pseudo boundaries and enhanced boundaries; the NSBG module is used to generate non-scene boundaries; the shot group in the sliding window randomly generates a shot sequence through the SBG module or the NSBG module as training data to achieve self-supervised video scene segmentation. The following is the working process of the two modules.
[0050] Step S2-1: Use the visual encoder to obtain i The features are extracted from the corresponding key frames and used as the features of the shot. The features of all shots construct the shot semantic feature sequence X = [x1, x2, ..., x t ], where x1,x2,xt They are lenses s1, s2, s t The lens feature, t is the length of the lens and lens semantic feature sequence;
[0051] Apply unsupervised time series segmentation to the extracted X to generate M consecutive and non-overlapping video clips, which form a pseudo scene set P = {p i |X=∪ i∈Μ p i}, where p i is the ith pseudo scene, a pseudo scene p i Corresponding to a video clip. Specifically, unsupervised time series segmentation is applied to the extracted shot semantic feature sequence X. The unsupervised time series segmentation algorithm can accept high-dimensional feature data and calculate the semantic relationship between sample points for segmentation. After the shot semantic feature sequence X is input into the unsupervised time series segmentation algorithm, the pseudo scene set P = {p i |X=∪ i∈Μ p i}, where p i is the ith pseudo scene.
[0052] Step S2-2: randomly process the video clips through SBG or NSBG to obtain several labeled B A and labeled N.
[0053] In SBG, a set of continuous shot segments B of length k is sampled in the pseudo scene using a sliding window z L , from B L Sampling shot fragment B of length k in the pseudo scene not in R , and B L and B R The pseudo scenes do not intersect, and B L May come from multiple pseudo scenes, B L and B R Spliced together, we get a set of synthetic shot sequences B, B with a length of 2k L and B R The shots at the splicing point are the boundaries of the synthetic video scene, and B is L and B R The shots in the scene are randomly exchanged to obtain the scene boundary shot sequence B after boundary enhancement A ; B L and B R The shot at the splicing point, that is, the middle shot of the shot sequence, is the pseudo-boundary shot, and this shot is labeled as a boundary.
[0054] In NSBG, a sliding window is used to sample N consecutive shots of length k in a pseudo scene. LIn order to ensure that the non-shot scene boundary shots and context shots have similar semantics, L Then continue to sample the shot sequence N with a length of k R It should be noted that: in a pseudo scene, a continuous shot N with a length of k is used L Then, we use a shot sequence N of length k. R If there are not enough shots in the pseudo scene, the missing shots are sampled from the next pseudo scene of the pseudo scene. L and N R Get the non-scene boundary shot sequence N. L and N R The shot at the splicing point, that is, the middle shot of the shot sequence, is a non-pseudo-boundary shot, and this shot is labeled as non-boundary.
[0055] Step S3: construct and train a boundary detection model, which includes a sequentially connected context encoder and a scene boundary classifier, a number of labeled B A and labeled N constitute a training set, one of which B A and N are each a training sample.
[0056] Step S3-1: Use the training samples in the training set to pre-train the boundary detection model to obtain a suboptimal boundary detection model; input the training samples into the boundary detection model, output the predicted label for each training sample, and use the binary cross entropy to calculate the training loss L p , as shown in the following formula:
[0057]
[0058] where f s (·) is the scene boundary classifier, It is B A The corresponding training samples after being encoded by the context encoder are is the training sample corresponding to N after being encoded by the context encoder.
[0059] The parameters of the boundary detection model are continuously updated according to the training loss until the maximum number of training times is reached or the loss stops decreasing, and the training is completed to obtain a suboptimal boundary detection model.
[0060] Step S3-2: Use the training samples in the training set to fine-tune the boundary detection model to obtain a suboptimal boundary detection model. Use the training samples to fine-tune the trained TSC, and use the following formula to calculate the fine-tuning loss L in the fine-tuning stage f :
[0061]
[0062] where fb (·) is the scene boundary classifier in the suboptimal boundary detection model, which needs to be trained from scratch. i is the true label of the training sample. The parameters of the suboptimal boundary detection model are continuously updated according to the loss until the maximum number of fine-tuning times is reached or the fine-tuning loss no longer decreases, and the fine-tuning is completed to obtain the optimal boundary detection model.
[0063] Step S4: Detection, such as Figure 2 As shown, the video clip to be detected is obtained through step S1 to obtain the shot sequence S', S' is input into the subsequent step S21 to obtain the feature sequence X', and the contextual association relationship between shots is constructed using a two-layer Transformer model. After X' is input into the two-layer Transformer, the 1024-dimensional shot feature with contextual information is output. A two-layer multi-layer perceptron (MLP) is used as the classifier of the final category of the shot on the shot feature with contextual information, and the label of the shot in the middle of the output sequence is the scene boundary.
[0064] Specifically, in step S2-1, a visual encoder is used to extract features from key frames to obtain features of each shot. The encoder for extracting shot features is obtained based on ResNet50 pre-trained on the ImageNet dataset. For S=[s1, s2, …, s t Every shot in i Three key frames f0, f1, and f2 are extracted to represent the shot. Each frame is passed through the feature encoder Enc to obtain 2048-dimensional features, and then a maximum pooling layer is passed to obtain the features of the shot:
[0065] x i =MaxPool(Enc[f0,f1,f2])
[0066] s i is the i-th lens, x i Represents the characteristics of the i-th lens.
[0067] Specifically, in step S2-2, unsupervised time series segmentation is applied to the extracted shot semantic feature sequence X. The unsupervised time series segmentation algorithm can accept high-dimensional feature data and calculate the semantic relationship between sample points for segmentation. After the shot semantic feature sequence X is input into the unsupervised time series segmentation algorithm, the pseudo scene set P = {p i |X=∪ i∈M p i}, where p i is the ith pseudo scene, M is the total number of pseudo scenes, and the number of scenes is calculated by the following formula:
[0068]
[0069] Where M is the number of target scenes, shot t is the total number of shots in the input video, and shot P is the number of shots in each pseudo scene in the current video, shot P Is a custom parameter.
[0070] Specifically, in step S2-2, if Figure 3 As shown in the left half, two video segments B of length k are selected from different pseudo scenes in P. L ={s l-k+1 ,...,s l} and B R ={s r ,...,s r+k-1}, and concatenate them to get B = {s l-k+1 ,...,s l ,s r ,...,s r+k-1}, it should be noted that B L With B R The pseudo scenes where the shots in B are located do not intersect, and L may come from multiple pseudo scenes, and in order to ensure low inter-scene similarity and high intra-scene similarity, B R Then start sampling from the first shot of a pseudo scene, so that B R More shots in the video come from the same pseudo scene. And P R =PP L .s r is a pseudo boundary shot. The scene boundary generation can be expressed as follows:
[0071]
[0072] Where b(·) extracts the boundary of the synthetic scene. L ={s i-k+1 ,...,s i} and B R ={s r ,...,s r+k-1 The shot features in} are calculated according to a certain number α and probability β e To exchange, such as Figure 4 As shown, the maximum number of exchange pairs is U, α is a random number between 1 and U, and β e is a fixed parameter, which indicates the probability of whether the shots in the synthesized scene obtained after the above steps have been completed are swapped.
[0073] The specific execution process of lens exchange is as follows:
[0074]
[0075] After exchanging the shots, we get the final scene boundary shot fragment B A ,Right now:
[0076]
[0077] The enhanced left half segment is represented as The right half is represented by Then the scene boundary enhancement can be expressed as follows:
[0078]
[0079] Where b(·) extracts the boundary of the synthetic scene. That is, the final pseudo scene boundary is s r .
[0080] Specifically, in step S2-2, a sliding window is used to sample continuous shots N with a length of k in a pseudo scene. L , N L ={s i-k+1 ,...,s i},like Figure 5 As shown, s i-k+1 is the first shot of the sliding window, s i is the last shot of the sliding window, when s i N L At the end of the pseudo scene, N R From i Start sampling and get N R ={s i ,...,s i+k-1};like Figure 3 As shown in the right half, when s i Not for N L At the end of the pseudo scene, N R Then, we can get N R ={s i+1 ,...,s i+k}; N L and N R The final non-scene boundary shot fragment N is obtained by splicing. Then N R It can be expressed as follows:
[0081]
[0082] Non-scene boundary generation can be expressed as follows:
[0083]
[0084] in Extract non-boundaries of synthetic scenes.
[0085] The core idea of the present invention is a method for constructing a pseudo-boundary of a scene. Specifically, two shot segments of the same length are sampled from the same video feature sequence and spliced together, and the two spliced shots are the synthesized video scene boundary. In order to make the scenes have a lower semantic similarity while maintaining a higher semantic similarity between the shots within the scene, the pseudo-scenes where the two selected segments are located do not intersect. In order to solve the lack of subdivision types in synthetic data, such as two adjacent scenes with similar elements, the present invention proposes to exchange the shots in the above two segments with a certain probability. Finally, the non-scene boundaries are classified and discussed to maintain semantic continuity to ensure that the obtained non-scene boundary shots have similar semantics to the context shots. A large number of experiments on very challenging data sets have verified that the method proposed in the present invention has good performance and advantages.
[0086] Experimental analysis
[0087] Table 1 Performance comparison of self-supervised transfer learning methods
[0088]
[0089]
[0090] The experimental data in Table 1 are the comparative results of self-supervised transfer learning experiments on the MovieNet dataset. This table compares several self-supervised transfer learning methods, and XFMR indicates that the contextual relationship between shots is encoded using a two-layer Transformer encoder. Performance on three evaluation indicators: AP, mIoU and F1. The present invention has achieved the most advanced results on multiple evaluation indicators. Compared with the ShotCoL method, the present invention has improved the AP and F1 evaluation indicators by 8.3% and 7.6% respectively. Compared with the BaSSL method, the present invention has improved the mIoU evaluation indicator by 8.1%. Compared with the CAT method, the present invention has improved the mIoU evaluation indicator by 5.0%. Compared with the most advanced TranS4mer method, the present invention has surpassed the TranS4mer method in all indicators, especially in the mIoU evaluation indicator by 6.7%. Among them, the BaSSL and TranS4mer methods are both pseudo-boundary prediction algorithms. The experimental results show that the performance of the present invention is superior to that of the pseudo-boundary prediction algorithm. These improvements verify the effectiveness and superiority of the method of the present invention.
[0091] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A self-supervised video scene boundary detection method based on a temporal scene creator, characterized in that: The process includes the following steps: Step S1: Extraction of shot sequence: Get an input video, get the shots in it, and form a shot sequence S = [s1, s2, ..., s t ], where s1, s2, s t The first shot, the second shot and the t-th shot are respectively sampled from the shots; Step S2: construct a temporal boundary creator network model TSC, TSC includes a parallel scene boundary generation module SBG and a non-scene boundary generation module NSBG; the SBG module is used to generate scene pseudo boundaries and enhanced boundaries; the NSBG module is used to generate non-scene boundaries; Step S2-1: Use the visual encoder to obtain i The features are extracted from the corresponding key frames and used as the features of the shot. The features of all shots construct the shot semantic feature sequence X = [x1, x2, ..., x t ], where x1,x2,x t They are lenses s1, s2, s t The lens feature, t is the length of the lens and lens semantic feature sequence; Apply unsupervised time series segmentation to the extracted X to generate M consecutive and non-overlapping video clips, which form a pseudo scene set P = {p i |X=∪ i∈M p i }, where p i is the ith pseudo scene, a pseudo scene p i Corresponding to a video clip; Step S2-2: randomly process the video clips through SBG or NSBG to obtain several labeled B A and labeled N. In SBG, a set of continuous shot segments B of length k is sampled in the pseudo scene using a sliding window z L , from B L Sampling shot fragment B of length k in the pseudo scene not in R , and B L and B R The pseudo scenes do not intersect, and B L May come from multiple pseudo scenes, B L and B R Spliced together, we get a set of synthetic shot sequences B, B with a length of 2k L and B R The shots at the splicing point are the boundaries of the synthetic video scene, and B is L and B R The shots in the scene are randomly exchanged to obtain the scene boundary shot sequence B after boundary enhancement A ; B L and B R The shots where they are spliced are marked with boundary labels. In NSBG, a sliding window is used to sample N consecutive shots of length k in a pseudo scene. L , in N L Then continue to sample the shot sequence N with a length of k R , splicing N L and N R Get the non-scene boundary shot sequence N. L and N R The shots at the splicing point are marked as non-boundary. Step S3: construct and train a boundary detection model, which includes a sequentially connected context encoder and a scene boundary classifier, a number of labeled B A and labeled N constitute a training set, one of which B A and B are each a training sample. Step S3-1: pre-training the boundary detection model using the training samples in the training set to obtain a suboptimal boundary detection model; The training samples are input into the boundary detection model, and the output is the predicted label of each training sample. The training loss L is calculated using binary cross entropy. p , as shown in the following formula: where f s (·) is the scene boundary classifier, It is B A The corresponding training samples after being encoded by the context encoder are is the training sample corresponding to N after being encoded by the context encoder; The parameters of the boundary detection model are continuously updated according to the training loss until the maximum number of training times is reached or the loss no longer decreases, and the training is completed to obtain a suboptimal boundary detection model. Step S3-2: Use the training samples in the training set to fine-tune the boundary detection model to obtain a suboptimal boundary detection model Use the training samples to fine-tune the trained TSC, and use the following formula to calculate the fine-tuning loss L in the fine-tuning stage f : where f b (·) is the scene boundary classifier in the suboptimal boundary detection model, which needs to be trained from scratch. i is the true label of the training sample. The parameters of the suboptimal boundary detection model are continuously updated according to the loss until the maximum number of fine-tuning times is reached or the fine-tuning loss no longer decreases, and the fine-tuning is completed to obtain the optimal boundary detection model. Step S4: Obtain the shot sequence S' of the segment to be detected through step S1, input S' into step S21 to obtain the feature sequence X', input X' into the optimal boundary detection model, and output a label indicating whether the middle shot of the video segment to be detected is a scene boundary.
2. A self-supervised video scene boundary detection method based on a temporal scene creator as claimed in claim 1, characterized in that: In step S2-2, obtain B A The steps include: Note B L ={s l-k+1 ,...,s l },s l-k+1 is the first shot of the sliding window, s l is the last shot in the sliding window, B R ={s r ,...,s r+k-1 },s r For B L A random shot in a pseudo-scene that is not there, s r+k-1 For r In the kth shot after that, B L and B R Splicing to get B = {s l-k+1 ,...,s l ,s r ,...,s r+k-1 }, the scene boundary generation is expressed as follows: Where b(·) extracts the boundary of the synthetic scene, and B L ={s l-k+1 ,...,s l } and B R ={s r ,...,s r+k-1 The shots in} are divided into the following order according to the number α and probability β e Exchange and get the final scene boundary shot fragment B A : B A The left half of The right half is represented by Then the scene boundary enhancement can be expressed as follows: s r is the final pseudo scene boundary.
3. A self-supervised video scene boundary detection method based on a temporal scene creator as claimed in claim 1 or 2, characterized in that: In step S2-2, the step of obtaining a non-scene boundary shot sequence N includes: Remember N L ={s i-k+1 ,...,s i },s i-k+1 is the first shot of the sliding window, s i is the last shot of the sliding window, when s i N L At the end of the pseudo scene, N R From i Start sampling and get N R ={s i ,...,s i+k-1 }; When s i Not for N L At the end of the pseudo scene, N R Then, we can get N R ={s i+1 ,...,s i+k }; N L and N R The final non-scene boundary shot fragment N is obtained by splicing, where s i+1 For i The next shot, s i+k For i After the k-th shot, N R It can be expressed as follows: Non-scene boundary generation can be expressed as follows: in Extract non-boundaries of synthetic scenes.
Citation Information
Patent Citations
Video scene segmentation method and device, visual task processing method and device, equipment and medium
CN115937742A
Cross-modal pedestrian re-identification method based on self-supervised learning and pre-training model
CN116052057A
Self-supervised video scene boundary detection method based on scene Mongolia
CN117058593A
Intelligent cataloging method for all-media news based on multi-modal information fusion understanding
US20220270369A1
