A video segmentation method based on Transformer
Through the video clip segmentation method based on Transformer, combined with video and voice information, intelligent segmentation and category annotation of videos are realized, solving the problem of inaccurate video clip segmentation in the prior art, and improving the accuracy and user experience of segmentation.
Patent Information
- Application Number
- CN202111120509.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-24
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-09-24
AI Technical Summary
The existing video clip segmentation method fails to effectively utilize the voice information in the video and the sequence information of the entire video, resulting in the segmented clips being trivial, the segmentation value is not high, and it is difficult to achieve precise positioning.
Using the video clip segmentation method based on Transformer, by sampling and annotating video and speech, a convolutional neural network and a Transformer model are constructed, image and speech features are extracted, position features are fused, model training and prediction are performed, and intelligent segmentation of video is realized. .
It realizes end-to-end video clip segmentation and category information provision, saves labor costs, enhances model robustness, can accurately express video information and build inter-frame relationships, and improves the accuracy and user experience of video segmentation.
Smart Images

Figure CN113837083B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of video segmentation, and in particular to a Transformer-based video segmentation method. Background Art
[0002] At present, videos are widely used as a promotional tool in the e-commerce field. For example, manufacturers’ display videos are played at cloud exhibitions. Buyers hope to accurately locate a certain segment of the video to view the information they need, such as the manufacturer’s qualifications or a product introduction. However, since the length and structure of each merchant’s video are different, it is difficult to segment the segments.
[0003] Traditional video segmentation mainly relies on the difference of video frame images for segmentation, and does not take into account the voice information in the video and the sequence information of the entire video. The segmented segments are relatively trivial and the segmentation value is not high.
[0004] Therefore, a segmentation method is needed that can combine the image and voice information in the video and the video sequence information, reasonably segment the video according to the combined information, and give specific categories for each segmented segment. Summary of the invention
[0005] The technical problem to be solved by the present invention is to overcome the deficiencies of the prior art and provide a video clip segmentation method based on Transformer.
[0006] In order to solve the above technical problems, the present invention provides a video segmentation method based on Transformer, which is characterized by sampling and labeling data sources, building and training models, and using models. The data sources include video and voice. The specific steps include:
[0007] Step S1, sampling and labeling the data source, adjusting the video duration to a preset duration, sampling the video screen to obtain a sampled image, sampling the video voice to obtain a sampled voice, matching the sampled image with the sampled voice and labeling the segmentation label;
[0008] Step S2: construct and train a transformer model, input the sampled image and the sampled speech into a convolutional neural network, the convolutional neural network includes a picture feature extraction network and a speech signal reading network, extract the convolution features of the sampled image and the MFCC features of the sampled speech, splice the convolution features of the sampled image and the MFCC features of the sampled speech, fuse the position features, obtain a corresponding sequence, and input the sequence into a preset transformer model for training;
[0009] Step S3: Use the transformer model. Process the target video through step S1, input the processing result into the transformer model trained in step S2, obtain the cutting prediction label of the target video, and segment the target video according to the cutting prediction label.
[0010] The step S1 specifically includes:
[0011] Step S1-1, adjusting the video duration to a preset duration includes: for a video shorter than the preset duration, using a blank screen and blank voice to complete it, for a video longer than the preset duration, using multiple speeds to compress it to the preset duration, and recording the compression multiples for restoring the position;
[0012] Step S1-2, sampling the video by frame sampling to obtain a sampled picture, sampling one picture per second, sampling the voice by second, and subdividing the sampled voice per second into a preset number of voice segments, corresponding to the sampled picture and the corresponding voice segment, to obtain a video segment with the sampled picture and the voice segment;
[0013] Step S1-3, annotating the video clips to obtain segmentation labels;
[0014] The step S2 specifically includes:
[0015] Step S2-4, scaling the sampled image to a preset fixed size, inputting the image feature extraction network, and extracting the corresponding convolution features;
[0016] Step S2-5, input the speech segment into the speech signal reading network to extract the corresponding MFCC features;
[0017] Step S2-6, concatenating the convolution features of the sampled image and the MFCC features of the speech clip, fusing the position features, and obtaining a sequence of data sources consisting of video clips;
[0018] Step S2-7, inputting the sequence into a preset transformer model, encoding and decoding the input through a multi-head encoder-decoder structure, and obtaining a feature vector for each video clip;
[0019] Step S2-8, sending the feature vector to the classification layer to obtain the corresponding predicted segmentation label, comparing the segmentation label marked in step S1-3, calculating the cross entropy loss between the two, and training the model through the neural network back propagation algorithm according to the principle of minimizing the loss, and updating the model parameters;
[0020] The step S3 specifically includes:
[0021] Step S3-9, passing the target video through steps S1-1 and S1-2 to obtain a data source containing video clips;
[0022] Step S3-10, inputting the corresponding sampled pictures and voice segments in the video segment into steps S2-4 and S2-5 respectively, to obtain corresponding convolution features and MFCC features;
[0023] Step S3-11, concatenate the convolution features of the sampled image and the MFCC features of the speech clip according to step S2-6, fuse the position features, input the trained and updated model parameters of the transformer model, obtain the cutting prediction label of the video clip, divide the data source according to the cutting prediction label and the compression rate of the video, and obtain the prediction results of the original video at each time point.
[0024] In step S1-1, the preset duration of the video is 15 minutes, and the preset number of paragraphs of the sampled speech per second is 40.
[0025] In the step S1-3, 1000 video clips are selected and segmented, and the video clip sequences are labeled with clip attributes according to categories, and the clip attributes include start, middle, and end.
[0026] In step S2-4, the preset fixed size is 224*224. In the image feature extraction network, Res2Net50 is selected as the backbone network, and a fully connected layer is added after the avgpooling layer of Res2Net50. The feature dimension of the fully connected layer is 512. The parameters of the image feature extraction network are fine-tuned using the pre-trained model of ImageNet data.
[0027] In the step S2-5, the MFCC feature extraction process is to perform high-pass filtering on the audio, then divide the audio into 40 frames per second, convert the audio signal from the time domain to the frequency domain through fast Fourier transform, take the modulus square of the audio signal spectrum to obtain the spectral line energy, and then obtain the Mel energy through Mel filtering, take the logarithm of the Mel energy and perform DCT transform to obtain the MFCC coefficient, retain the 13-dimensional coefficient of the MFCC coefficient, perform first-order difference and second-order difference on the MFCC coefficient, and obtain 39-bit features for each frame of audio.
[0028] In step S2-6, the position feature is encoded in advance, and the advance encoding adopts the sin-cos rule. The feature vector of the sequence extracted in step S2-6 and the position feature of the same size are added and fused to obtain the feature vector of the sequence for inputting the transformer model.
[0029] In step S2-8, the features of each frame of video are obtained through transformer, and a fully connected layer is constructed, the output dimension of which is the total number of all label categories, and softmax is used to output the predicted label with the highest probability corresponding to each frame of video.
[0030] The beneficial effects achieved by the present invention are:
[0031] 1. It can realize end-to-end segmentation of video clips and provide segment category information, which is convenient for users watching videos to quickly locate them. It replaces manual segmentation work and saves labor costs.
[0032] 2. Use a pre-trained image feature extraction model and add a fully connected layer to achieve model parameter reuse and reduce the feature dimension of the image, so that lower-dimensional features can well represent the image and enhance the robustness of the model.
[0033] 3. Use image features, speech features and video sequence features to express video information more comprehensively, which helps to accurately segment the video.
[0034] 4. By using the transformer model to model the sequence, we can build relationships at any position and realize the joint learning of information between frames. We can classify the frames through the classification layer, and provide the category information of the segmented segments while segmenting the video, so that users can directly jump to the search.
[0035] 5. Use the trained model to predict new videos, and manually correct and add new data to the model training set to ensure that the model maintains high segmentation accuracy and good robustness. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] Figure 1 A schematic diagram of a method flow of an exemplary embodiment of the present invention;
[0037] Figure 2 Schematic diagram of data and model structure in an exemplary embodiment of the present invention. DETAILED DESCRIPTION
[0038] The present invention will be further described below in conjunction with the accompanying drawings and exemplary embodiments:
[0039] like Figure 1 As shown, a video segmentation method based on transformer in this embodiment, transformer is a model based on encoder-decoder structure, mainly comprising the following steps:
[0040] Step S1: Align the training data, unify the videos to 15 minutes, sample 900 frames, and perform sequence labeling on the 900 frames, labeling the current frame category, including the category name and its position.
[0041] Step S2: 900 frames of data are used for feature extraction. Res2Net50 is used to extract 512-dimensional features for images, and MFCC is used to extract 40 segments corresponding to each frame, totaling 1560-dimensional features, and the image features are spliced with the speech features.
[0042] Step S3: Generate video sequence position features to obtain a position feature vector of size (900, 2072), which is added and fused with the video image and speech features of size (900, 2072) generated in step S2 to obtain a video sequence feature of size (900, 2072) for the entire video.
[0043] Step S4: Build a transformer model using a standard multi-head attention structure. The number of heads is set to 6. The encoder and decoder are set to 6 layers. The decoded output is followed by a fully connected layer with the total number of category and position labels. The label with the highest probability is output through softmax and compared with the manually labeled label in step S1. The loss is calculated and the model parameters are updated.
[0044] Step S5: Except for the annotations in step S1, the new video is processed in the same way as steps S1-S3, and the model trained in step S4 is input to obtain the specific category and position of each frame. According to the video rate ratio, the position is restored in the original video, and the recognition is divided into segments of different categories.
[0045] In step S1, 1000 videos are selected as training data. Since the lengths of videos vary, videos with a duration of less than 15 minutes are completed to 15 minutes by using blank frames and blank voices, and videos with a duration of more than 15 minutes are compressed to 15 minutes by doubling the speed.
[0046] In step S1, a 15-minute video is sampled, one frame of image and one second of voice are sampled per second, and a total of 900 frames of data are obtained. The 900 frames of data are labeled, and the labels are a combination of category + position, for example, the beginning of the product segment is PB, the end of the product segment is PE, and the middle of the product segment is PO.
[0047] In step S2, the video image is scaled to 224*224, Res2Net50 is selected as the feature extractor, a fully connected layer is added after the avgpooling layer of Res2Net50, 512 is selected for the fully connected layer, and other structures remain unchanged. The parameters are fine-tuned using the pre-trained model on the ImageNet data, and the image features are extracted using the trained model, and the feature dimension is 512.
[0048] In step S2, the audio per second is divided into 40 segments, each segment is 25ms. The power spectrum is estimated by using the periodogram method for each segment, the power spectrum is filtered by the Mel filter bank, the energy in each filter is calculated, the energy of each filter is logarithmized, the DCT transformation is performed, the 2nd to 13th coefficients of the DCT are retained, the others are removed, the energy and 12 coefficients are used as basic features, and the 13-dimensional features are first-order and second-order differences are performed to obtain a total of 39 audio MFCC features, and the 40 audio segments have a total of 1560 dimensions.
[0049] In step S2, the image features and the speech features are concatenated to obtain a 2072-bit feature vector for each frame.
[0050] In step S3, since the transformer adopts a parallel architecture, it has no ability to capture sequential sequences. However, in the video segmentation task, the sequence position is particularly important. Therefore, for the video feature sequence, it is necessary to add position features. The position feature is encoded in advance, using the sin-cos rule. pos corresponds to a specific frame position in 900, i corresponds to the corresponding dimension in 2072, and d corresponds to the total dimension of the vector 2072. The 2072 dimensions of the vector are divided into odd rows and even rows. The even rows are encoded using the sin function, and the odd rows are encoded using the cos function. Finally, the position feature of (900, 2072) is obtained. The sin-cos rule is as follows:
[0051] PE(pos,2i)=sin(pos / 10000 2i / d )
[0052] PE(pos,2i+1)=cos(pos / 10000 2i / d );
[0053] The image speech features and position features of the same dimension are added and fused to obtain the sequence features of the entire video.
[0054] In step S4, the entire video training feature is input into the transformer model, and a new vector is output after passing through the encoder and decoder. A fully connected layer is connected after the decoder layer, and the size is the total number of labels in step S1. The vector of each frame is output to the label with the maximum probability through softmax, and compared with the manually labeled label in step S1, the loss is calculated, and the model parameters are updated. The loss function is the cross entropy loss, and the optimizer is Adam.
[0055] In step S5, for a new video, no labeling is required, and preprocessing is continued according to steps S1-S3, and the model trained in step S4 is input to obtain predicted labels for each frame of the video, and the video is divided into different segments according to the predicted labels.
[0056] Figure 2 It is a schematic diagram of data and model structure of a transformer-based video clip segmentation method of the present invention.
[0057] Structure 1 is the input of the transformer model, which is generated by steps S1-S3. The video is aligned to 900 seconds, 900 frames are sampled, and image features and speech features are extracted using image feature extractors and MFCCs respectively. After concatenation, position features are added to finally obtain the sequence features of the entire video as input. The input size is (900, 2072).
[0058] Structure 2 is the main structure of the transformer model. The model contains 6 encoder modules and 6 decoder modules. Each module is a standard 12-head attention structure. Multi-head attention can be used to focus on information in different subspaces and achieve information fusion.
[0059] Structure 3 is the decoded output of the transformer model. Through the encoder and decoder modules, the previously output (900, 2072)-dimensional vector is encoded and decoded into a (900, 2072)-dimensional vector. After encoding and decoding, the vector fuses the global features and obtains the information of each position.
[0060] Structure 4 is a fully connected layer, which maps the 2072-dimensional feature vector in structure 3 to the space of N category positions, outputs the category probability through softmax, obtains the category with the highest probability, and converts the vector into a specific category position. N is the segmentation label defined previously, including category and position.
[0061] Structure 5 is the output label, with a size of (900,1), that is, each frame outputs a category position label, through which the video can be divided into different segments.
[0062] The present invention is mainly used to provide a transformer-based video segmentation method, which uses an artificial intelligence deep learning model to achieve significant results in the following aspects:
[0063] 1. It can realize end-to-end segmentation of video clips and provide segment category information, which is convenient for users watching videos to quickly locate them. It replaces manual segmentation work and saves labor costs.
[0064] 2. Use a pre-trained image feature extraction model and add a fully connected layer to achieve model parameter reuse and reduce the feature dimension of the image, so that lower-dimensional features can well represent the image and enhance the robustness of the model.
[0065] 3. Use image features, speech features and video sequence features to express video information more comprehensively, which helps to accurately segment the video.
[0066] 4. By using the transformer model to model the sequence, we can build relationships at any position and realize the joint learning of information between frames. We can classify the frames through the classification layer, and provide the category information of the segmented segments while segmenting the video, so that users can directly jump to the search.
[0067] 5. Use the trained model to predict new videos, and manually correct and add new data to the model training set to ensure that the model maintains high segmentation accuracy and good robustness.
[0068] The present invention makes full use of the multimodal features of images and audio in the video and uses the advantages of the transformer model structure in modeling long sequences to transform the segmentation of video clips into a sequence labeling problem, thereby further segmenting the video into multiple clips through the sequence labeling results.
[0069] The above embodiments do not limit the present invention in any way. Any other improvements and applications made to the above embodiments in an equivalent transformation manner belong to the protection scope of the present invention.
Claims
1. A video segmentation method based on Transformer, characterized in that: Sampling and labeling data sources, building and training models, and using models. The data sources include video and voice. The specific steps include: Step S1, sampling and labeling the data source, adjusting the video duration to a preset duration, sampling the video screen to obtain a sampled image, sampling the video voice to obtain a sampled voice, matching the sampled image with the sampled voice and labeling the segmentation label; Step S2: construct and train a transformer model, input the sampled image and the sampled speech into a convolutional neural network, the convolutional neural network includes a picture feature extraction network and a speech signal reading network, extract the convolution features of the sampled image and the MFCC features of the sampled speech, splice the convolution features of the sampled image and the MFCC features of the sampled speech, fuse the position features, obtain a corresponding sequence, and input the sequence into a preset transformer model for training; Step S3: Use the transformer model to process the target video in step S1, input the processing result into the transformer model trained in step S2, obtain the cutting prediction label of the target video, and segment the target video according to the cutting prediction label; The step S1 specifically includes: Step S1-1, adjusting the video duration to a preset duration includes: for a video shorter than the preset duration, using a blank screen and blank voice to complete it, for a video longer than the preset duration, using a multiple speed to compress it to the preset duration, and recording the compression multiple rate for restoring the position; Step S1-2, sampling the video by frame sampling to obtain a sampled picture, sampling one picture per second, sampling the voice by second, and subdividing the sampled voice per second into voice segments of a preset number of paragraphs, corresponding to the sampled picture and the corresponding voice segment, to obtain a video segment with the sampled picture and the voice segment; Step S1-3, annotating the video clips to obtain segmentation labels; The step S2 specifically includes: Step S2-4, scaling the sampled image to a preset fixed size, inputting the image feature extraction network, and extracting the corresponding convolution features; Step S2-5, input the speech segment into the speech signal reading network to extract the corresponding MFCC features; Step S2-6, concatenating the convolution features of the sampled image and the MFCC features of the speech clip, fusing the position features, and obtaining a sequence of data sources consisting of video clips; Step S2-7, inputting the sequence into a preset transformer model, encoding and decoding the input through a multi-head encoder-decoder structure, and obtaining a feature vector for each video clip; Step S2-8, sending the feature vector to the classification layer to obtain the corresponding predicted segmentation label, comparing the segmentation label marked in step S1-3, calculating the cross entropy loss between the two, and training the model through the neural network back propagation algorithm according to the principle of minimizing the loss, and updating the model parameters; The step S3 specifically includes: Step S3-9, passing the target video through steps S1-1 and S1-2 to obtain a data source containing video clips; Step S3-10, inputting the corresponding sampled pictures and voice segments in the video segment into steps S2-4 and S2-5 respectively, to obtain corresponding convolution features and MFCC features; Step S3-11, concatenate the convolution features of the sampled image and the MFCC features of the speech segment according to step S2-6, fuse the position features, input the trained and updated model parameters of the transformer model, obtain the cutting prediction label of the video segment, divide the data source according to the cutting prediction label and the compression rate of the video, and obtain the prediction results of each time point of the original video; In step S1-1, the preset duration of the video is 15 minutes, and the preset number of paragraphs of the sampled speech per second is 40; In the step S1-3, 1000 video clips are selected and segmented, and the video clip sequences are labeled with clip attributes according to categories, and the clip attributes include start, middle, and end.
2. The method for segmenting a video clip based on Transformer according to claim 1, characterized in that: In step S2-4, the preset fixed size is 224*224. In the image feature extraction network, Res2Net50 is selected as the backbone network, and a fully connected layer is added after the avgpooling layer of Res2Net50. The feature dimension of the fully connected layer is 512. The parameters of the image feature extraction network are fine-tuned using the pre-trained model of ImageNet data.
3. The method for segmenting a video clip based on Transformer according to claim 2, characterized in that: In the step S2-5, the MFCC feature extraction process is to perform high-pass filtering on the audio, then divide the audio into 40 frames per second, convert the audio signal from the time domain to the frequency domain through fast Fourier transform, take the modulus square of the audio signal spectrum to obtain the spectral line energy, and then obtain the Mel energy through Mel filtering, take the logarithm of the Mel energy and perform DCT transform to obtain the MFCC coefficient, retain the 13-dimensional coefficient of the MFCC coefficient, perform first-order difference and second-order difference on the MFCC coefficient, and obtain 39-bit features for each frame of audio.
4. The method for segmenting a video segment based on Transformer according to claim 3, characterized in that: In step S2-6, the position feature is encoded in advance, and the advance encoding adopts the sin-cos rule. The feature vector of the sequence extracted in step S2-6 and the position feature of the same size are added and fused to obtain the feature vector of the sequence for inputting the transformer model.
5. The method for segmenting a video segment based on Transformer according to claim 4, characterized in that: In step S2-8, the features of each frame of video are obtained through transformer, and a fully connected layer is constructed, the output dimension of which is the total number of all tag categories, and softmax is used to output the predicted label with the highest probability corresponding to each frame of video.
