A method and system for detecting highlight shots with improved feature extraction of large models

Abstract

The application discloses a highlight shot detection method and system for improving feature extraction of a large model. The method comprises the following steps: inputting a video to be processed, extracting video frames at a certain time interval, preprocessing the extracted video frames to obtain preprocessed video frames; inputting the preprocessed video frames into a text image language large model BLIP2 to obtain a text description of the video frames; inputting the preprocessed video frames and the text description of the video frames into a pre-trained image text model CLIP to extract overall image features and overall text features, and fusing the overall image features and the overall text features to obtain fused features X; calculating a shot highlight score by using the fused features X, and inputting the shot highlight score into a loss function to train a model; and inputting a video and a query statement into the trained model to obtain a highlight video segment related to the query statement in the video. The application can reduce the dependence on artificial labeled data, improve the quality of feature extraction, thereby improving the accuracy and efficiency of video content understanding and retrieval, and improving the generalization of the method.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video understanding, and in particular to a highlight shot detection method and system for improving feature extraction of large models. BACKGROUND

[0002] In recent years, video has become an important carrier for people to convey information. The massive amount of video brings people visual enjoyment, but also carries a large amount of redundant information. People's desire to filter out redundant information and quickly find the highlights in the video has spurred and promoted the research of video highlight detection task. The goal of the video highlight detection task is to quickly retrieve the relevant highlight video segments in the given video when a user query statement is given. In order to achieve this goal, it is necessary to effectively model the association between video and query text. However, video and query text are two different forms of information carriers. How to make these two different information carriers interact fully and efficiently complete the video highlight detection task is a key challenge.

[0003] The current mainstream video moment retrieval method is to use a deep neural network to model the interaction between video and query text, and to train using the corresponding artificially annotated dataset. However, this method usually requires very high human cost, and it is difficult to obtain sufficient training data. Therefore, some methods use pre-trained image-text models (such as CLIP) to extract video frame features and text features, and then input these features into a small model and train using a small amount of artificially annotated data, in order to utilize the existing mature image-text data knowledge and improve the accuracy of video highlight detection. Later, large models gradually matured, but most of them are natural language large models, only having a decoder but lacking a suitable encoder to obtain pre-trained features.

[0004] One of the current prior arts is the patent "Video important shot detection method and system based on cross-modal reasoning (CN118053096A)". This method first encodes and fuses the visual, audio and text information of the video, and uses an attention mechanism to capture the context information, and then outputs the importance score of each video segment through a multi-layer perception. The cross-modal fusion module of this method can effectively utilize multi-modal information and improve the accuracy and comprehensiveness of video important shot detection. However, the disadvantage of this method is that it fails to utilize the prior knowledge of existing large models, and there is still room for improvement, and its general applicability is poor, and it can only be applied to the video highlight detection task and has multiple alternative solutions. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the prior art, and a highlight shot detection method and system for improving feature extraction of large models are proposed. The main problem solved by the present application is how to reduce the dependence on manual annotation data, improve the quality of feature extraction, thereby improving the accuracy and efficiency of video content understanding and retrieval, and improving the generalization of the method.

[0006] To solve the above problems, the present application proposes a highlight shot detection method for improving feature extraction of large models, which comprises:

[0007] Input the video to be processed, extract video frames at a certain time interval, so that the video frames can effectively represent the features in that period of time, and perform preprocessing operations on the extracted video frames to obtain preprocessed video frames, denoted as N.

[0008] Input N preprocessed video frames into a text image language large model BLIP2 to obtain N groups of text descriptions of video frames;

[0009] Input N preprocessed video frames and N groups of text descriptions of video frames into a pre-trained image text model CLIP to extract overall image features and overall text features, fuse the overall image features and overall text features to obtain fusion features X R N×D , wherein D is the dimension of a single feature;

[0010] Calculate the highlight score of the shot using the fusion features X, input the highlight score of the shot into a loss function to train the model, and obtain the trained model;

[0011] The user inputs the video and the query statement into the trained model to obtain the highlight video segment related to the query statement in the video.

[0012] Preferably, the preprocessing operation is specifically:

[0013] Adjust the pixel size of the video frame to 224x224, and normalize the video frame after adjusting the pixel size, so that the RGB value range of each pixel point is [0, 1]

[0014] Preferably, the input of N preprocessed video frames and N groups of text descriptions of video frames into the pre-trained image text model CLIP to extract overall image features and overall text features, and the fusion of the overall image features and the overall text features to obtain the fusion features X R N×D is specifically:

[0015] Input N preprocessed video frames into a pre-trained image text model CLIP, and for each preprocessed video frame, obtain a one-dimensional vector v i R 1×DAs the video image features of the video frames, i∈[1,N],for all the extracted video frames, the overall image features V∈R N×D ;

[0016] The N groups of text descriptions of the video frames are input into a pre-trained image text model CLIP, and for each group of text descriptions of the video frames, a plurality of one-dimensional text features are obtained, wherein the first one-dimensional text feature is designated as the overall feature t of the group of text descriptions i ∈R 1×D The overall features of the N groups of text descriptions are spliced to obtain overall text features, denoted as T∈R N×D ;

[0017] The overall image features and the overall text features are fused according to the following formula:

[0018] X=Norm(V+T,dim=-1),

[0019] Wherein, Norm represents normalization, dim=-1 represents normalization only in the last dimension, that is, the mean μ and the variance σ of the last dimension of the fusion feature are calculated, and each value of the last dimension of the fusion feature is subtracted from the mean μ and divided by the variance σ.

[0020] Preferably, the lens highlight score S is calculated using the fusion feature X, the lens highlight score is input into a loss function to train the model, and a trained model is obtained, specifically:

[0021] A linear mapping network Linear is used to map the fusion feature X to a dimension of 1, and then a Sigmoid function is used to map the value to [0,1] to obtain N lens highlight scores S∈R N×1 As shown in the following formula:

[0022] S=Sigmoid(Linear(X)),

[0023] Let a lens highlight score be s i , i∈[1,N],the model is trained using the cross-entropy loss function using the lens highlight score S, and the real label is g i , i∈[1,N],then the cross-entropy loss function MSE is specifically:

[0024]

[0025] Correspondingly, the application also provides a highlight lens detection system for improving feature extraction of a large model, comprising:

[0026] A preprocessing unit is configured to input a video to be processed, extract video frames at a certain time interval, make the video frames effectively represent the features in the time period, and perform a preprocessing operation on the extracted video frames to obtain preprocessed video frames, wherein the number of the preprocessed video frames is denoted as N; and input the N preprocessed video frames into a text-image language large model BLIP2 to obtain N groups of text descriptions of the video frames;

[0027] A feature extraction and fusion unit is configured to input the N preprocessed video frames and the N groups of text descriptions of the video frames into a pre-trained image-text model CLIP, extract overall image features and overall text features, fuse the overall image features and the overall text features to obtain fusion features X e R N×D , wherein D is the dimension of a single feature.

[0028] A shot highlight score calculation and model training unit is configured to calculate a shot highlight score by using the fusion features X, input the shot highlight score into a loss function to train a model, and obtain a trained model.

[0029] An application unit is configured to input a video and a query statement into the trained model by a user to obtain a highlight video segment related to the query statement in the video.

[0030] The present application has the following beneficial effects:

[0031] The video highlight shot detection method based on description enhancement can significantly improve the accuracy and efficiency of video content understanding and retrieval. The present application automatically generates description texts of video frames by using a natural language processing large model, extracts features in combination with a pre-trained image-text model (such as CLIP), reduces the dependence on a large amount of manually annotated data, reduces the cost in the data preparation stage, improves the quality and expression ability of the features because of the fusion of visual and semantic information, and enables the model to better understand the video content, thereby enhancing the performance of video highlight shot detection. In addition, the flexibility of the present application allows it to be easily integrated into existing video analysis frameworks, providing new research directions and technical means for the field of video content analysis, and further promoting the progress of automated video understanding and processing. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 is a flowchart of a highlight shot detection method of a large model improved feature extraction according to an embodiment of the present application;

[0033] Figure 2 is a structural diagram of a highlight shot detection system of a large model improved feature extraction according to an embodiment of the present application. DETAILED DESCRIPTION

[0034] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0035] Figure 1 is a flowchart of a highlight shot detection method for feature extraction of a large model according to an embodiment of the present application, as shown in Figure 1 , the method comprises:

[0036] S1, input a video to be processed, extract video frames at a certain time interval, so that the video frames can effectively represent the features in the time period, and perform a preprocessing operation on the extracted video frames to obtain preprocessed video frames, denoted as N.

[0037] S2, input N preprocessed video frames into a text image language large model BLIP2 to obtain N groups of text descriptions of video frames.

[0038] S3, input N preprocessed video frames and N groups of text descriptions of video frames into a pre-trained image text model CLIP to extract overall image features and overall text features, fuse the overall image features and the overall text features to obtain fusion features X R N×D , where D is the dimension of a single feature.

[0039] S4, calculate the highlight score of the shot using the fusion features X, input the highlight score of the shot into a loss function to train the model, and obtain a trained model.

[0040] S5, a user inputs a video and a query statement into the trained model to obtain a highlight video segment related to the query statement in the video.

[0041] Step S1 is as follows:

[0042] S1-1, the time interval for extracting video frames is 2 seconds, and the tool used for extracting video frames is the ffmpeg tool, which can extract video frames according to the time interval and save them as images.

[0043] S1-2, using the torchvision library to adjust the pixel size of the video frame to 224x224, and normalizing the video frame after adjusting the pixel size, so that the RGB value range of each pixel point is [0, 1], wherein the normalization can directly call the Normalize function in the torch framework, and the function parameters are set to mean (0.485, 0.456, 0.406), variance (0.229, 0.224, 0.225). torchvision and torch are both machine learning related libraries in python.

[0044] Step S2 is as follows:

[0045] The text image language model BLIP2 extracts the video frame features and generates text descriptions through its decoding ability to convert visual language into natural language.

[0046] The text description usually contains detailed explanations of objects, scenes and possible activities in the image.

[0047] Step S3 is as follows:

[0048] S3-1, input N preprocessed video frames into the pre-trained image text model CLIP, for each of the preprocessed video frames, obtain a one-dimensional vector v i ∈R 1×D As the video image feature of the video frame, i∈[1,N], then for all extracted video frames, the overall image feature V∈R N×D ;

[0049] S3-2, each group of text description of the video frame contains several description texts of different lengths, which are converted into several tokens in natural language processing, therefore, input the text description of N groups of video frames into the pre-trained image text model CLIP, to obtain several one-dimensional text features, wherein the first one-dimensional text feature is designated as the overall feature t i ∈R 1×D of the group of text descriptions, the overall features of N groups of text descriptions are spliced to obtain the overall text feature, denoted as T∈R N×D , the overall text feature T and the overall image feature V are one-to-one corresponding;

[0050] S3-3, since the overall text feature and the overall image feature are of the same size and one-to-one corresponding, they can be directly added and normalized, that is, the feature fusion is completed, and the overall image feature and the overall text feature are fused according to the following formula:

[0051] X=Norm(V+T,dim=-1),

[0052] wherein Norm denotes normalization, dim = -1 denotes normalization only in the last dimension, i.e. calculating the mean mu and variance sigma of the last dimension of the fusion feature, and subtracting the mean mu and dividing by the variance sigma from each value of the last dimension of the fusion feature.

[0053] Step S4 is specifically as follows:

[0054] S4-1, using a linear mapping network Linear, the fusion feature X is mapped to dimension 1, and then the value is mapped to [0, 1] through a Sigmoid function, to obtain N highlight scores S e R N×1 as shown in the following formula:

[0055] S = Sigmoid(Linear(X)),

[0056] S4-2, a highlight score is recorded as s i , i e [1, N], using the highlight score S to train the model using a cross-entropy loss function, and recording the true label as g i , i e [1, N], then the cross-entropy loss function MSE is specifically:

[0057]

[0058] Correspondingly, the application also provides a highlight shot detection system for improving feature extraction of a large model, as shown in Figure 2 , comprising:

[0059] A preprocessing unit 1 is configured to input a video to be processed, extract video frames at a certain time interval, make the video frames effectively represent the features in the time period, perform a preprocessing operation on the extracted video frames, obtain preprocessed video frames, and record the number of the preprocessed video frames as N; input the N preprocessed video frames into a text image language large model BLIP2 to obtain N groups of text descriptions of the video frames.

[0060] Specifically, the preprocessing operation includes: adjusting the pixel size of the video frame to 224x224, and normalizing the video frame after adjusting the pixel size, so that the RGB value of each pixel point is in the range of [0, 1].

[0061] A feature extraction and fusion unit 2 is configured to input the N preprocessed video frames and the N groups of text descriptions of the video frames into a pre-trained image text model CLIP to extract overall image features and overall text features, fuse the overall image features and the overall text features, and obtain a fusion feature X e R N×D , wherein D is the dimension of a single feature.

[0062] Specifically, N pre-processed video frames are input into a pre-trained image-text model CLIP, and for each of the pre-processed video frames, a one-dimensional vector v is obtained i ∈R 1×D As the video image features of the video frame, i∈[1,N], for all the extracted video frames, the overall image features V∈R can be obtained N×D ;

[0063] N groups of text descriptions of the video frames are input into the pre-trained image-text model CLIP, and for each group of text descriptions of the video frames, a plurality of one-dimensional text features are obtained, wherein the first one-dimensional text feature is designated as the overall feature t of the group of text descriptions i ∈R 1×D The overall features of the N groups of text descriptions are spliced to obtain the overall text features, denoted as T∈R N×D ;

[0064] The overall image features and the overall text features are fused according to the following formula:

[0065] X=Norm(V+T,dim=-1),

[0066] Where Norm represents normalization, and dim=-1 represents normalization only in the last dimension, that is, the mean μ and the variance σ of the last dimension of the fusion feature are calculated, and each value of the last dimension of the fusion feature is subtracted from the mean μ and divided by the variance σ.

[0067] The shot highlight score calculation and model training unit 3 is used to calculate the shot highlight score using the fusion feature X, input the shot highlight score into a loss function to train the model, and obtain the trained model;

[0068] Specifically, a linear mapping network Linear is used to map the fusion feature X to a dimension of 1, and then a Sigmoid function is used to map the value to [0, 1] to obtain N shot highlight scores S∈R N×1 As shown in the following formula:

[0069] S=Sigmoid(Linear(X)),

[0070] Let a shot highlight score be s i , i∈[1,N], the model is trained using the cross-entropy loss function using the shot highlight score S, and the real label is g i , i∈[1,N], then the cross-entropy loss function MSE is specifically:

[0071]

[0072] The application unit 4 is configured to input the video and the query statement into the trained model by a user to obtain a highlight video clip related to the query statement in the video.

[0073] Therefore, the present application proposes a highlight shot detection method for large model improved feature extraction. This method can significantly improve the accuracy and efficiency of video content understanding and retrieval. This method automatically generates description text for video frames by using natural language processing large models, and combines pre-trained image-text models (such as CLIP) to extract features, which not only reduces the dependence on a large amount of manually annotated data and reduces the cost of the data preparation stage, but also improves the quality and expression ability of the features because of the fusion of visual and semantic information. High-quality features enable the model to better understand video content, thereby enhancing the performance of video highlight shot detection. In addition, the flexibility of this method makes it easy to integrate into existing video analysis frameworks, providing new research directions and technical means for the field of video content analysis, and further promoting the progress of automated video understanding and processing.

[0074] The above describes a highlight shot detection method and system for large model improved feature extraction provided by the embodiment of the present application in detail. The principle and implementation mode of the present application are described by applying specific examples in this paper. The above description of the embodiments is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed; in summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A highlight shot detection method for improving feature extraction of a large model, characterized in that, The method comprises: inputting a video to be processed, extracting video frames at a certain time interval, performing a preprocessing operation on the extracted video frames to obtain preprocessed video frames, and recording the number of the preprocessed video frames as N; inputting the N preprocessed video frames into a text image language large model BLIP2 to obtain N sets of text descriptions of the video frames; inputting N pieces of the preprocessed video frames and N sets of the text descriptions of the video frames into a pre-trained image-text model CLIP, extracting overall image features and overall text features, fusing the overall image features and the overall text features to obtain fused features X e R N×D wherein D is the dimension of a single feature; calculating a highlight score of a shot using the fusion feature X, inputting the highlight score of the shot into a loss function to train the model, and obtaining a trained model; a user inputs a video and a query statement into the trained model to obtain a highlight video segment related to the query statement in the video; Specifically, the N pre-processed video frames and the N sets of text descriptions of the video frames are input into a pre-trained image-text model CLIP to extract overall image features and overall text features, the overall image features and the overall text features are fused to obtain a fusion feature X∈R N×D , specifically: inputting N pieces of the preprocessed video frames into a pre-trained image-text model CLIP, obtaining a one-dimensional vector v for each of the preprocessed video frames i ∈R 1×D As the video image features of the video frames, i∈[1, N], for N pieces of the preprocessed video frames, a total image feature V∈R N×D ; inputting the text descriptions of the N groups of video frames into a pre-trained image-text model CLIP, obtaining a plurality of one-dimensional text features for each group of text descriptions of the video frames, wherein the first one-dimensional text feature is designated as the overall feature t of the text descriptions of the group of video frames i ∈R 1×D splicing the overall features of the N groups of text descriptions of the video frames to obtain an overall text feature, denoted as T∈R N×D ; fusing the overall image feature and the overall text feature according to the following formula: X=Norm(V+T,dim=-1), wherein Norm represents normalization, and dim=-1 represents normalization only in the last dimension, that is, calculating the mean μ and the variance σ of the last dimension of the fusion feature, and subtracting the mean μ and dividing by the variance σ of each value of the last dimension of the fusion feature.

2. The method of claim 1, wherein the method is improved by a large model. The preprocessing operation specifically comprises: adjusting the pixel size of the video frame to 224×224, and normalizing the video frame after adjusting the pixel size, so that the RGB value range of each pixel point is [0, 1].

3. The method of claim 1, wherein the method is improved by a large model. The calculation of the highlight score S of the shot using the fusion feature X, the inputting of the highlight score of the shot into the loss function to train the model, and the obtaining of the trained model specifically comprise: The fusion feature X is mapped to dimension 1 using a linear mapping network Linear, and then the value is mapped to [0, 1] through a Sigmoid function to obtain N shot exciting scores S∈R N×1 as shown in the following formula: S=Sigmoid(Linear(X)), A shot highlight score is denoted as s i , i ∈ [1, N], the model is trained using the cross-entropy loss function with the shot highlight score S, and the real label is denoted as g i , i ∈ [1, N], and the cross-entropy loss function MSE is specifically:

4. A highlight shot detection system that improves feature extraction of a large model, characterized by, The system comprises: a preprocessing unit configured to input a video to be processed, extract video frames at a certain time interval, perform a preprocessing operation on the extracted video frames to obtain preprocessed video frames, record the number of the preprocessed video frames as N, and input the N preprocessed video frames into a text image language large model BLIP2 to obtain N sets of text descriptions of the video frames; The feature extraction and fusion unit is configured to input N preprocessed video frames and N sets of text descriptions of the video frames into a pre-trained image-text model CLIP to extract overall image features and overall text features, fuse the overall image features and the overall text features, and obtain a fusion feature XeR N×D where D is the dimension of a single feature. a shot highlight score calculation and model training unit configured to calculate a highlight score of a shot using a fusion feature X, input the highlight score of the shot into a loss function to train the model, and obtain a trained model; an application unit configured to input a video and a query statement into the trained model by a user to obtain a highlight video segment related to the query statement in the video; Specifically, the N pre-processed video frames and the N sets of text descriptions of the video frames are input into a pre-trained image-text model CLIP to extract overall image features and overall text features, the overall image features and the overall text features are fused to obtain a fusion feature X∈R N×D , specifically: inputting N pieces of the preprocessed video frames into a pre-trained image-text model CLIP, obtaining a one-dimensional vector v for each of the preprocessed video frames i ∈R 1×D As the video image features of the video frames, i∈[1,N], for N pieces of the preprocessed video frames, a total image feature V∈R N×D ; inputting the text descriptions of the N groups of video frames into a pre-trained image-text model CLIP, obtaining a plurality of one-dimensional text features for each group of text descriptions of the video frames, wherein the first one-dimensional text feature is designated as the overall feature t of the text descriptions of the group of video frames i ∈R 1×D splicing the overall features of the N groups of text descriptions of the video frames to obtain an overall text feature, denoted as T∈R N×D ; fusing the overall image feature and the overall text feature according to the following formula: X=Norm(V+T,dim=-1), wherein Norm represents normalization, and dim=-1 represents normalization only in the last dimension, that is, calculating the mean μ and the variance σ of the last dimension of the fusion feature, and subtracting the mean μ and dividing by the variance σ of each value of the last dimension of the fusion feature.

5. The system for highlight shot detection with improved feature extraction of a big model as claimed in claim 4, wherein, The preprocessing operation in the preprocessing unit specifically comprises: adjusting the pixel size of the video frame to 224×224, and normalizing the video frame after adjusting the pixel size, so that the RGB value range of each pixel point is [0, 1].

6. The system for highlight shot detection with improved feature extraction of large models as claimed in claim 4 wherein, The calculation of the highlight score S of the shot using the fusion feature X, the inputting of the highlight score of the shot into the loss function to train the model, and the obtaining of the trained model specifically comprise: The fusion feature X is mapped to dimension 1 using a linear mapping network Linear, and then the value is mapped to [0, 1] through a Sigmoid function to obtain N shot exciting scores S∈R N×1 As shown in the following formula: S=Sigmoid(Linear(X)), A shot highlight score is denoted as s i , i ∈ [1, N], the model is trained using the cross-entropy loss function with the shot highlight score S, and the real label is denoted as g i , i ∈ [1, N], and the cross-entropy loss function MSE is specifically: 7.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-6 when the computer program is executed by the processor. The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 3.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 3.

Images