A video scene boundary detection method

Abstract

The present invention relates to a scene boundary detection method for a video. The method guides a model to learn contextual semantic relations between shots by generating a pseudo scene boundary. The training of a neural network model includes two stages, a pre-training stage and a fine-tuning stage. In the pre-training, the neural network model is pre-trained by learning pseudo scene boundary predictions similar to scene boundary predictions. Subsequently, the pre-trained model is fine-tuned by real scene boundary annotation information, so as to obtain good scene boundary prediction results in real data.

Description

Technical Field

[0001] The present invention belongs to the field of deep learning in artificial intelligence, and specifically is a method for discovering potential patterns of semantic transitions in video scenes and detecting scene boundaries through video self-supervision. Background Art

[0002] Among the self-supervised learning methods currently used for video scene boundary detection, one type focuses on learning video representations and uses contrastive learning methods to learn representations that are useful for video scene boundary detection tasks; the other type considers the relationship between shots and pre-trains the model by designing a pseudo boundary prediction task as a proxy task. The pseudo boundary prediction task has similar training objectives to the target task (video scene boundary detection task), and thus shows great potential. The pseudo boundary prediction task relies on the pseudo boundary generation method, but the discussion of the relationship between shots in existing work is not comprehensive enough, and it is impossible to generate high-quality pseudo scene boundaries to effectively guide the model to learn contextual semantic relationships, resulting in a large difference between the generated pseudo boundaries and the real scene boundaries. Summary of the invention

[0003] In view of the above problems existing in the prior art, the technical problem to be solved by the present invention is: how to improve the quality of the generated pseudo scene boundary.

[0004] To solve the above technical problems, the present invention adopts the following technical solution: a method for detecting scene boundaries of a video, comprising the following steps:

[0005] S1: Extraction of shot sequence: Get an input video, get the shots in it, and form a shot sequence S = [s1,…s n ] to sample key frames from the shot.

[0006] S2: Build and train a scene boundary detection neural network.

[0007] S21: Use the visual encoder to extract features from the key frames to obtain the features of each shot. The features of all shots construct a feature sequence X = [x1,…,x n ], where one of the features x i Corresponding to a lens s i , n is the length of the shot and feature sequence.

[0008] S22: Set two movable anchor points along the shot sequence to divide the shot sequence into two pseudo scenes S left ={s0,…,s b-1} and S right ={s b ,…,s n}, the last shot in the previous pseudo scene is an anchor point, and the first shot in the next pseudo scene is another anchor point. The similarity scores of the anchor points in one pseudo scene and all the shots in another pseudo scene are calculated.

[0009] S23: normalize the similarity scores based on the anchor points, and normalize the similarity obtained in S22.

[0010] S24: Make the anchor point traverse the entire shot sequence, calculate the similarity results of all possible pseudo scenes, and compare S left and S right By traversing the offset b, we can find the pseudo scene boundary b that can minimize the similarity between scenes. * .

[0011] S25: Take the feature sequence X corresponding to the shot sequence of S21 as input, and use the pseudo scene boundary and non-pseudo scene boundary information generated by S24 as supervision signals to complete the pre-training of the scene boundary detection neural network.

[0012] S26: Use the real annotated scene boundary information to fine-tune the pre-trained scene boundary detection neural network obtained in S25 to obtain the final video scene boundary detection model.

[0013] S3: Detection. The video clip to be detected is used to obtain a shot sequence S' through step S1, and then a corresponding feature sequence X' is obtained through step S21. X' is input into the final video scene boundary detection model, and the confidence that the middle shot of the sequence is a scene boundary is output.

[0014] As an improvement, the specific method for calculating the semantic similarity between an anchor point in a pseudo scene and all shots in another pseudo scene in S22 is: b-1 and x b is the anchor point, by calculating the anchor point lens x b-1 With S right , x b With S left The similarity scores of all shots between the two pseudo scenes are obtained by:

[0015]

[0016] Among them, f STD (b) indicates that the lens s b is the semantic similarity between two pseudo scenes when it is the pseudo scene boundary.

[0017] The similarity score between shots is expressed by the cosine similarity between the features corresponding to the shots:

[0018]

[0019] Among them, x i Indicates lens i The corresponding feature, x j Indicates lens j Corresponding features.

[0020] As an improvement, the specific steps of normalizing the similarity score based on the anchor point in S23 are as follows: by calculating the similarity score between the anchor point and all the shot features in the entire shot sequence, summing the scores and taking the square root as the normalization coefficient of the similarity, the normalization formula is:

[0021]

[0022] Among them, w a，j Represents anchor point s α With lens j The standardized similarity between .

[0023] As an improvement, the pseudo scene boundary b in S24 * The calculation formula is:

[0024]

[0025] Among them, b * is the pseudo scene boundary, w b，i Represents the anchor lens s b With lens i The standardized similarity w b-1，j Represents the anchor lens s b-1 With lens j The standardized similarity.

[0026] As an improvement, the specific steps of pre-training the scene boundary detection neural network in S25 are as follows:

[0027] Apply position information to each element in X so that the context encoder can perceive the order of elements. The position encoding is essentially another feature vector sequence P = [p1, ..., p n ], feed the sequence X with position information into the context encoder to obtain a new feature vector sequence R = [r1, ..., r n ]:

[0028] R = Context(concat(P, X))

[0029] Among them, Context(·) is the context encoder and concat(·) is the vector concatenation operation.

[0030] Take each element in R as a training sample and assign a label to each element. The starting shot is s b The elements in R corresponding to the other shots are regarded as positive samples and assigned the label y = 1, while the elements in R corresponding to other shots are regarded as negative samples and assigned the label y = 0; thus, the pseudo boundary prediction task is transformed into a shot classification task, where the feature vector r b Corresponding positive sample lens

[0031] Then randomly select an element r from R i As the feature vector corresponding to the negative sample lens, finally, the feature vectors corresponding to the positive and negative samples Input to a multilayer perceptron and pre-train the context encoder by minimizing the binary cross entropy loss:

[0032]

[0033] Among them, h p (·) is a multilayer perceptron with two linear layers and one normalization layer.

[0034] As an improvement, the specific steps of fine-tuning the pre-trained scene boundary detection neural network in S26 are as follows:

[0035] For a video segment S = [s1, ...s n ], the duration of each shot is encoded and concatenated together to form a new feature vector sequence D = [d1, ..., d n ]:

[0036] D = f Linear ([δ1,…,δ n ])

[0037] f Linear (·) represents the duration encoder;

[0038] For a video segment S = [s1, ...s n ], the input sequence of the hybrid encoder consists of a feature vector sequence R = [r1, ..., r n ], a vector sequence D containing shot duration information = [d1, ..., d n ], and the position code P = [p1, ..., p n ], the output sequence E = [e1,…,e n ] is the result of mixing the two modes and interacting with each other in context:

[0039] E = Hybrid (concat (R, P, D))

[0040] Where Hybrid(·) is a hybrid encoder, and the central lens s c The corresponding mixed eigenvector e c is fed into a multilayer perceptron h f In (·), the pre-trained scene boundary detection neural network is fine-tuned by minimizing the binary cross entropy loss:

[0041]

[0042] Among them, h f (·) is the same as h p (·) Another multilayer perceptron with the same structure, if the lens s c Located in a real view

[0043] The starting position of the video scene, then its label yc = 1, otherwise y c =0.

[0044] Compared with the prior art, the present invention has at least the following advantages:

[0045] 1. Different from the traditional method of selecting anchor points at both ends of a shot sequence to calculate the similarity with non-anchor shots, the present invention selects two adjacent shots as anchor points to calculate the similarity with shots in another pseudo scene, which is more in line with the definition of the similarity relationship between scenes. Based on this method, all possible pseudo scenes are traversed, and the results are more accurate.

[0046] 2. By normalizing the similarity based on anchor points, the interference of unclear semantics between shots caused by shooting techniques on the similarity calculation between shots is reduced.

[0047] 3. Through a new inter-scene similarity calculation strategy, the two pseudo-scenes obtained have lower inter-scene semantic similarity than the traditional method, thereby obtaining a pre-training supervision signal with clearer and more accurate features, and ultimately improving the model's ability to discern scene boundaries. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 The video scene boundary detection task is regarded as a shot binary classification task.

[0049] Figure 2 For the pre-training workflow.

[0050] Figure 3 Sets the structure diagram for the anchor point.

[0051] Figure 4 The structure and workflow of the hybrid encoder.

[0052] Figure 5 For fine-tuning your workflow. DETAILED DESCRIPTION

[0053] The present invention is described in further detail below.

[0054] The present invention regards video scene boundary detection as Figure 1 The shot binary classification task shown in the figure is to determine whether each shot is located at the starting position of a scene. A video scene is composed of one or more complete shots, and each shot can be regarded as a basic unit in the scene. Therefore, detecting scene boundaries in a video is essentially to determine whether the temporal boundaries between adjacent shots are the temporal boundaries between adjacent scenes.

[0055] Since the boundary of the scene is exactly the boundary of the shot, the video scene boundary detection can be transformed into a binary classification task for the shot, that is, each shot is divided into two categories: shots at the beginning of the scene and shots at other positions in the scene. If the current shot is judged to be the beginning of the scene, then the previous shot is also considered to be the end of the scene. By classifying each shot, the model can detect the scene boundary.

[0056] In order to segment a long video into scenes, we first need to decompose the video into a shot sequence S consisting of continuous and non-overlapping shots. video =[s1,…s i …s N ],s i represents the i-th shot.

[0057] The method of the present invention first designs a pseudo boundary generation method, namely, a semantic transition detection method (STD). This method divides a video segment into two pseudo scenes by detecting the semantic transition between shots. Then, a context encoder is pre-trained using the pseudo boundary between pseudo scenes, and the context encoder can learn the relationship between a shot and its context shot. After completing the pre-training of the context encoder, the model is fine-tuned in combination with the real scene boundary annotation and the introduction of shot duration information to adapt to the video scene boundary detection task.

[0058] A method for detecting scene boundaries of a video comprises the following steps:

[0059] S1: Extraction of shot sequence: Get an input video, get the shots in it, and form a shot sequence S = [s1,…s n ] to sample key frames from the shot.

[0060] S2: Construct and train a scene boundary detection neural network, wherein the scene boundary detection neural network includes a visual encoder, a context encoder, a multi-layer perceptron, a temporal encoder, and a hybrid encoder.

[0061] In the self-supervised video scene boundary detection task, pseudo-label refers to the scene boundary generated by the method on the unlabeled video, which is used to replace the real scene boundary annotation for model training. Pseudo-label is also called pseudo-boundary in the present invention. By assuming that a video segment is a scene, the video segment can be called a pseudo-scene. Pseudo-boundary is the time boundary of the pseudo-scene. Its quality directly affects the model learning effect. How to generate high-quality pseudo-boundary is one of the important issues of the present invention. Pseudo-boundary prediction refers to the proxy task of training the video scene boundary detection model by predicting pseudo-boundary in the absence of annotation. Pseudo-boundary prediction can help the model model the relationship between different shots, so as to more confidently judge the scene boundary and obtain more accurate scene boundary detection results.

[0062] The present invention uses pseudo boundary prediction as a pre-training task to improve the performance of video scene boundary detection. This method can weaken the dependence of the model's learning process on manually labeled scene boundaries, and use unlabeled data for self-supervised learning to learn representations useful for video scene boundary detection. At the same time, it aims to narrow the difference between pseudo labels and true labels to capture more accurate supervisory signals, thereby learning high-quality contextual representations. By increasing the similarity between pseudo labels and true labels, the information in unlabeled data can be better utilized, thereby improving the performance of the model.

[0063] A STD method based on SI relations is designed, which generates pseudo boundaries by focusing on the semantic inconsistency between each shot and its context. Before introducing the pseudo boundary generation method, a sliding window is first introduced to solve the problem of too long videos.

[0064] In specific implementation, the size of the sliding window can be fixed to n shots, and the step length can be 1 shot. Then each video segment contains n shots, and the long video S containing N shots in total video It can generate N-n+1 video segments. Using the sliding window method to decompose a long video into several short videos is an effective processing method, which can help the method process video data more easily and provide the possibility for efficient generation of pseudo boundaries. For a decomposed video segment containing n shots.

[0065] S21: Use the visual encoder to extract features from the key frames to obtain the features of each shot. The features of all shots construct a feature sequence X = [x1,…,x n ], where one of the features x i Corresponding to a lens s i , n is the length of the shot and feature sequence.

[0066] S22: See Figure 3 , set two movable anchor points along the shot sequence, and divide the shot sequence into two pseudo scenes S left={s0, ..., s b-1} and S right ={s b , ..., s n}, the last shot in the previous pseudo scene is an anchor point, and the first shot in the next pseudo scene is another anchor point. The similarity scores of the anchor points in one pseudo scene and all the shots in another pseudo scene are calculated.

[0067] S23: normalize the similarity scores based on the anchor points, and normalize the similarity obtained in S22.

[0068] S24: Make the anchor point traverse the entire shot sequence, calculate the similarity results of all possible pseudo scenes, and compare S left and S right By traversing the offset b, we can find the pseudo scene boundary b that can minimize the similarity between scenes. * : The anchor point position with the lowest semantic similarity between scenes is retrieved among all the standardized similarities of a single feature sequence, and the semantic transition position is used as the boundary of the pseudo scene.

[0069] S25: Take the feature sequence X corresponding to the shot sequence of S21 as input, and use the pseudo scene boundary and non-pseudo scene boundary information generated by S24 as supervision signals to complete the pre-training of the scene boundary detection neural network.

[0070] S26: Use the real annotated scene boundary information to fine-tune the pre-trained scene boundary detection neural network obtained in S25 to obtain the final video scene boundary detection model.

[0071] S3: Detection, obtain the shot sequence S' of the video segment to be detected through step S1, and then obtain the corresponding feature sequence X' through step S21, input X' into the final video scene boundary detection model, and output the confidence that the middle shot of the sequence is the scene boundary. The long video is segmented by assigning a confidence score to each shot. After applying the sliding window to the long video segmented into a shot sequence, the model gives a confidence score between 0 and 1 to the center shot in each video segment. When the confidence score is greater than 0.5, the shot is determined to be the starting shot of a scene, that is, the left time boundary of the shot is determined to be a scene boundary. When the sliding window traverses the long video, the scene boundary detection of the entire long video is completed. Specifically, the specific method for calculating the semantic similarity between an anchor point in a pseudo-scene and all shots in another pseudo-scene in S22 is:

[0072] x b-1 and x b is the anchor point, by calculating the anchor point lens x b-1 With S right, x b With S left The similarity scores of all shots between the two pseudo scenes are obtained by STD (b)

[0073]

[0074] Among them, f STD (b) indicates that the lens s b is the semantic similarity between two pseudo scenes when it is the pseudo scene boundary.

[0075] Shot Sequence S video =[s1, ...s N ] is divided into two pseudo scenes S left ={s0, ..., s b-1} and S right ={s b , ..., s n},s b-1 and b They are pseudo scenes S left The last shot and pseudo scene S right The starting shot is s. b With S left The shots in the should be different, and the shots s b-1 With S right The shots in should be dissimilar. By traversing b and adding s b-1 and b As a pair of anchor points, used to represent the video segment S video =[s1, ...s N ] generates a pseudo scene boundary. Thus, the STD method can compare adjacent scenes S on the lens left and S right Similarity between.

[0076] The similarity score between shots is expressed by the cosine similarity between the features corresponding to the shots:

[0077]

[0078] Among them, x i Indicates lens i The corresponding feature, x j Indicates lens j Corresponding features.

[0079] Specifically, the specific steps of normalizing the similarity score based on the anchor point in S23 are as follows: by calculating the similarity score between the anchor point and all the shot features in the entire shot sequence, the square root is taken after the sum is taken as the normalization coefficient of the similarity, so as to alleviate the influence of abnormal shots introduced by shooting techniques on the similarity calculation. The normalization formula is:

[0080] When the anchor point itself is an abnormal shot, the STD method can always obtain a lower inter-scene similarity score, thus misjudging the shot of the abnormal shot.

[0081] The boundary is a pseudo-scene boundary. In this case, the robustness of the STD method to abnormal shots needs to be improved. In order to solve this problem, this section designs an anchor-based normalization (ABN) method. The main idea of ​​the ABN method is to normalize the similarity score of the anchor point to a range, which is calculated by the similarity score of the anchor point shot and multiple shots near it. Specifically, in a given video segment S = [s1,…s n ], for any anchor point lens s n , all the shots in S can be regarded as shots in its neighborhood. ABN will first calculate the anchor shot s n The similarity score with each shot in S is calculated, and the bi-norm of the score is calculated. This value is then used to normalize the similarity score between the anchor point and each shot in the neighborhood so that the score falls within a certain range.

[0082]

[0083] Among them, w a,j Represents anchor point s a With lens j The standardized similarity between .

[0084] Specifically, the specific steps of calculating the similarity results of all possible pseudo-scenes in S24 are as follows:

[0085] Specifically, the pseudo scene boundary b in S24 * The calculation formula is:

[0086]

[0087] Among them, b * is the pseudo scene boundary, w b,i Represents the anchor lens s b With lens i The standardized similarity, w b-1,j Represents the anchor lens s b-1 With lens jThe meaning of this is that at this point, there is a minimum semantic similarity between the two pseudo scenes.

[0088] For details, see Figure 2 The specific steps of pre-training the scene boundary detection neural network in S25 are as follows:

[0089] In the processing of long videos, how to analyze the relationship between a shot and its temporal context is a key issue. The contextual information of a shot can give the elements in the shot a more complete semantics, thereby helping the model to analyze and understand the video. In order to solve the problem of modeling the relationship between shots, this section first builds a network model suitable for processing sequence data, and then introduces how to use the pseudo labels generated by the STD method to train the model.

[0090] Apply position information to each element in X so that the context encoder can perceive the order of the elements. The position information is obtained by concatenating trainable position encodings, which are essentially another feature vector sequence P = [p1, ..., p n ], feed the sequence X with position information into the context encoder to obtain a new feature vector sequence R = [r1, ..., r n ]:

[0091] R = Context(concat(P, X))

[0092] Among them, Context(·) is the context encoder and concat(·) is the vector concatenation operation.

[0093] By performing S2 semantic transition detection on X, we can obtain the pseudo scene boundary b * , and determine the element For pseudo scene S right Medium starting shots b The corresponding feature vector, since the present invention regards the video scene boundary detection task as a binary classification task, that is, to determine any shot s b Is it the starting shot? Therefore, each element in R is used as a training sample and a label is assigned to each element. The starting shot s b The elements in R corresponding to the other shots are regarded as positive samples and assigned the label y = 1, while the elements in R corresponding to other shots are regarded as negative samples and assigned the label y = 0; thus, the pseudo boundary prediction task is transformed into a shot classification task, where the feature vector r b Corresponding positive sample lens

[0094] In order to balance the number of positive and negative samples, randomly select an element r from R iAs the feature vector corresponding to the negative sample lens, finally, the feature vectors corresponding to the positive and negative samples The input is fed into a multi-layer perceptron (MLP) to pre-train the context encoder by minimizing the binary cross entropy loss:

[0095]

[0096] Among them, h p (·) is a multilayer perceptron with two linear layers and one normalization layer.

[0097] Specifically, the specific steps of fine-tuning the pre-trained scene boundary detection neural network in S26 are as follows:

[0098] In order to train a complete model for video scene boundary detection, a new module needs to be designed to integrate the context encoder and the duration encoder, and the context encoder needs to be fine-tuned using video segments with real labels.

[0099] The integration of context encoder and duration encoder needs to consider the structure and meaning of their output data. The role of context encoder is to give context information to the feature vector of each shot, and the role of duration encoder is to encode the duration of the shot into a feature vector. The sequence structure composed of the two types of feature vectors is similar and the meaning is uncoupled, so the two feature vectors can be directly spliced ​​to fuse the information, and the sequence composed of the spliced ​​feature vectors can be fed into a simple encoder for further context interaction.

[0100] The duration of a shot is a real number in seconds. The duration encoder uses a linear mapping to convert the duration of each shot δ i Encoded as a feature vector d i .

[0101] For a video segment S = [s1, ...s n ], the duration of each shot is encoded and concatenated together to form a new feature vector sequence D = [d1, ..., d n ]:

[0102] D = f Linear ([δ1,…,δ n ])

[0103] f Linear (·) represents the duration encoder.

[0104] In order to achieve contextual interaction between feature vectors that mix the two modalities, this section introduces a hybrid encoder constructed by a single-layer Transformer encoder.

[0105] For a video segment S = [s1, ...s n ], the input sequence of the hybrid encoder consists of a feature vector sequence R = [r1, ..., r n ], a vector sequence D containing shot duration information = [d1, ..., d n ], and the position code P = [p1, ..., p n ], the output sequence E = [e1,…,e n ] is the result of mixing the two modes and interacting with each other in context:

[0106] E = Hybrid (concat (R, P, D))

[0107] Among them, Hybrid(·) is a hybrid encoder, and its workflow is as follows Figure 4 shown.

[0108] For the video segment S' with real scene annotations, with the help of the hybrid encoder, the duration encoder can be fine-tuned together with the pre-trained context encoder. For the video segment S' with real scene annotations, through the collaboration of the visual encoder, duration encoder, context encoder and hybrid encoder, a feature vector sequence E that integrates all information can be obtained, where the first element e c Corresponding to the shot s at the center of the sequence in the video segment S' c , compared to the other lenses in S', the central lens s c The lower limit of the number of preceding and following shots is the highest, so it has the most favorable conditions to obtain the reorganized context information.

[0109] The workflow of the fine-tuning phase is as follows: Figure 5 In order to uniformly train the context encoder, duration encoder, and hybrid encoder, the center shot s c The corresponding mixed eigenvector e c is fed into a multilayer perceptron h f In (·), the pre-trained scene boundary detection neural network is fine-tuned by minimizing the binary cross entropy loss:

[0110]

[0111] Among them, h f (·) is the same as h p (·) Another multilayer perceptron with the same structure, if the lens s c Located in a real view

[0112] The starting position of the video scene, then its label y c =1, otherwise y c =0.

[0113] Example:

[0114] Taking the MovieNet dataset as an example, the dataset provides the shot information of the video and the key frames in the shot. The pre-trained ResNet50 model is used as a single-frame visual feature extractor, and the features of three key frames are used to represent the features of the corresponding shot.

[0115] In the sequence of all shots in a single video, it is necessary to extract multiple shot subsequences S = [s1, ..., s n ], corresponding to the characteristic subsequence X = [x1, ..., x n ], the length n of the sequence used in the present invention is 13.

[0116] After obtaining the feature sequence, semantic transition detection is required, that is, through the anchor point s b-1 With s b Split the feature sequence X into two pseudo scenes S left ={s0,…,s b-1} and S right ={s b ,…,s n}, we can know that the first anchor point lens s b-1 Belongs to the first pseudo scene S left , the second anchor point lens s b Belongs to the second pseudo scene S right .

[0117] The similarity calculation formula is used to calculate the similarity between each anchor point and all the lens features in another pseudo scene to represent the semantic relationship between them.

[0118] When calculating the semantics between an anchor point and a shot, it is necessary to calculate the feature similarity between the anchor point shot and the entire shot sequence, sum them up and then square them to obtain the standardized parameters based on the anchor point, and finally obtain the s a With lens j The standardized similarity w between a,j .

[0119] After completing the calculation of all standardized similarities, the semantic transition detection formula, that is, the pseudo scene boundary calculation formula, is used. A value b is found, where the semantic association between the two pseudo scenes is minimal, and the position in the sequence corresponding to the value is the semantic transition position.

[0120] Sampling is performed on each feature sequence, and the shot features at the pseudo-boundary are assigned a "1" label to represent the scene boundary, and the shot features at the non-pseudo-boundary are assigned a "0" label to represent the non-scene boundary. The minimum number of shot samples at the non-boundary in a single sequence is 1.

[0121] Construct a neural network consisting of a two-layer Transformer as a contextual relationship modeling module and a two-layer MLP as a boundary detection module. Set the learning rate of the pre-training phase to 0.3, and use the SGD optimizer with stochastic gradient descent.

[0122] The obtained feature sequence is used as the input of the context semantic relationship modeling neural network to obtain the feature sequence of semantic relationship modeling. Feature filtering is performed based on the sampling information, and only the features at the sampling point are retained and input into the scene boundary detection module to obtain the boundary prediction results of each input shot.

[0123] The binary cross entropy is used as the loss calculator for the boundary prediction results. The loss is calculated based on the labels assigned during sampling, and then the gradient information is updated through backpropagation. After 21 cycles of training, the pre-trained scene boundary detection model is obtained.

[0124] The performance comparison of the pre-trained scene boundary detection model of the present invention and the existing self-supervised method training model on the MovieNet test set is shown in the following table:

[0125] Table 1: Comparison of accuracy of self-supervised models

[0126] method AP QUR AUC-ROC F1 StoryGraph 25.10 35.70 - - Grouping 33.60 37.20 - - Our invention 37.22 39.58 78.24 34.35

[0127] After completing the self-supervised pre-training, the annotated data in the MovieNet data is organized into the same format. The starting shot of the scene is labeled with a "1" label to indicate the scene boundary, and the remaining shots are labeled with a "0" label to indicate non-scene boundaries. The training subset is used as the training data for the neural network.

[0128] Set the learning rate of the fine-tuning phase to 10 -5 , and uses the Adam optimizer.

[0129] During the fine-tuning stage, a shot duration encoder consisting of a linear layer is added to the model. After encoding the duration information of each shot, it is added with the position encoding and spliced ​​onto the contextual relationship features of the corresponding shot to obtain a mixed feature.

[0130] The mixed features are input into a hybrid encoder consisting of a layer of Transformer, and then the shot features output by the hybrid encoder are input into the MLP to obtain the boundary prediction result.

[0131] Use binary cross entropy as the loss calculation function to calculate the loss between the scene boundary prediction result and the actual labeled boundary label, and complete the model update after backpropagating the gradient information; train for 21 cycles to obtain the final fine-tuning model.

[0132] The fine-tuned model is used to verify the model's scene segmentation capability on the MovieNet validation set. The comparison results with other existing methods are shown in Table 2:

[0133] Table 2: Final accuracy results of self-supervised training on the MovieNet validation dataset

[0134] method AP QUR ROC F1 SCRL 54.55 - - 51.39 BaSSL 57.40 50.69 90.54 47.02 Ours 60.61 51.72 90.97 48.26

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

Claims

1. A method for detecting scene boundaries in a video, characterized in that: The steps include: S1: Extraction of shot sequence: Get an input video, get the shots in it, and form a shot sequence S = [s1, ...s n ], sampling key frames from the shot; S2: Build and train the scene boundary detection neural network and training; S21: Use the visual encoder to extract features from the key frames to obtain the features of each shot. The features of all shots construct a feature sequence X = [x1, ..., x n ], where one of the features x i Corresponding to a lens s i , n is the length of the lens and feature sequence; S22: Set two movable anchor points along the shot sequence to divide the shot sequence into two pseudo scenes S left ={s0, ..., s b-1 } and S right ={s b , ..., s n }, the last shot in the previous pseudo scene is an anchor point, and the first shot in the next pseudo scene is another anchor point. The similarity scores of the anchor points in one pseudo scene and all the shots in another pseudo scene are calculated; S23: normalize the similarity scores based on the anchor points, and normalize the similarity obtained in S22; S24: Make the anchor point traverse the entire shot sequence, calculate the similarity results of all possible pseudo scenes, and compare S left and S right By traversing the offset b, we can find the pseudo scene boundary b that can minimize the similarity between scenes. * ; S25: Taking the feature sequence X corresponding to the shot sequence in S21 as input, and using the pseudo scene boundary and non-pseudo scene boundary information generated in S24 as supervisory signals, complete the pre-training of the scene boundary detection neural network; S26: fine-tune the pre-trained scene boundary detection neural network obtained in S25 using the real annotated scene boundary information to obtain a final video scene boundary detection model; S3: Detection. The video clip to be detected is used to obtain a shot sequence S' through step S1, and then a corresponding feature sequence X' is obtained through step S21. X' is input into the final video scene boundary detection model, and the confidence that the middle shot of the sequence is a scene boundary is output.

2. The method for detecting scene boundaries of a video according to claim 1, wherein: The specific method for calculating the semantic similarity between an anchor point in a pseudo scene and all shots in another pseudo scene in S22 is: b-1 and x b is the anchor point, by calculating the anchor point lens x b-1 With S right , x b With S left The similarity scores of all shots between the two pseudo scenes are obtained by: Among them, f STD (b) indicates that the lens s b The semantic similarity between two pseudo scenes when it is the pseudo scene boundary; The similarity score between shots is expressed by the cosine similarity between the features corresponding to the shots: Among them, x i Indicates lens i The corresponding feature, x j Indicates lens j Corresponding features.

3. The method for detecting scene boundaries of a video according to claim 2, wherein: The specific steps of normalizing the similarity score based on the anchor point in S23 are as follows: by calculating the similarity score between the anchor point and all the shot features in the entire shot sequence, summing the scores and taking the square root as the normalization coefficient of the similarity, the normalization formula is: Among them, w a，j Represents anchor point s a With lens j The standardized similarity between .

4. The method for detecting scene boundaries of a video according to claim 1, wherein: The pseudo scene boundary b in S24 * The calculation formula is: Among them, b * is the pseudo scene boundary, w b，i Represents the anchor lens s b With lens i The standardized similarity w b-1，j Represents the anchor lens s b-1 With lens j The standardized similarity.

5. The method for detecting scene boundaries of a video according to claim 1, wherein: The specific steps of pre-training the scene boundary detection neural network in S25 are as follows: Apply position information to each element in X so that the context encoder can perceive the order of elements. The position encoding is essentially another feature vector sequence P = [p1, ..., p n ], feed the sequence X with position information into the context encoder to obtain a new feature vector sequence R = [r1, ..., r n ]: R = Context(concat(P, X)) Among them, Context(·) is the context encoder, and concat(·) is the vector concatenation operation; Take each element in R as a training sample and assign a label to each element. The starting shot is s b The elements in R corresponding to the other shots are regarded as positive samples and assigned the label y = 1, while the elements in R corresponding to the other shots are regarded as negative samples and assigned the label y = 0; thus, the pseudo boundary prediction task is transformed into a shot classification task, where the feature vector r b Corresponding positive sample lens Then randomly select an element r from R i As the feature vector corresponding to the negative sample lens, finally, the feature vectors corresponding to the positive and negative samples Input to a multilayer perceptron and pre-train the context encoder by minimizing the binary cross entropy loss: Among them, h p (·) is a multilayer perceptron with two linear layers and one normalization layer.

6. The method for detecting scene boundaries of a video according to claim 1, wherein: The specific steps of fine-tuning the pre-trained scene boundary detection neural network in S26 are as follows: For a video segment S = [s1, ...s n ], the duration of each shot is encoded and concatenated together to form a new feature vector sequence D = [d1, ..., d n ]: D=f Linear ([δ1,…,δ n ]) f Linear (·) represents the duration encoder; For a video segment S = [s1, ...s n ], the input sequence of the hybrid encoder consists of a feature vector sequence R = [r1, ..., r n ], a vector sequence D containing shot duration information = [d1, ..., d n ], and the position code P = [p1, ..., p n ], the output sequence E = [e1,…,e n ] is the result of mixing the two modes and interacting with each other in context: E = Hybrid (concat (R, P, D)) Where Hybrid(·) is a hybrid encoder, and the central lens s c The corresponding mixed eigenvector e c is fed into a multilayer perceptron h f In (·), the pre-trained scene boundary detection neural network is fine-tuned by minimizing the binary cross entropy loss: Among them, h f (·) is the same as h p (·) Another multilayer perceptron with the same structure, if the lens s c Located at the starting position of a real video scene, its label y c =1, otherwise y c =0.

Images