Large model improved feature extraction-based wonderful lens detection method and system

Abstract

The invention discloses a wonderful lens detection method and system for large model improved feature extraction. Comprising the following steps: inputting a to-be-processed video, extracting video frames according to a certain time interval, and preprocessing the extracted video frames to obtain preprocessed video frames; inputting the video frame into a text image language large model BLIP2 to obtain text description of the video frame; inputting the preprocessed video frame and the text description of the video frame into a pre-training image text model CLIP, extracting an overall image feature and an overall text feature, and fusing the overall image feature and the overall text feature to obtain a fused feature X; calculating a shot wonderful score by using the fusion feature X, and inputting the shot wonderful score into a loss function to train the model; and the user inputs the video and the query statement into the trained model to obtain a wonderful video clip related to the query statement in the video. According to the method, the dependence on manual annotation data can be reduced, the quality of feature extraction is improved, the accuracy and efficiency of video content understanding and retrieval are improved, and the generalization of the method is improved.

Description

Technical Field

[0001] The present invention relates to the field of video understanding, and in particular to a wonderful shot detection method and system for large-model improved feature extraction. Background Art

[0002] In recent years, video has become an important carrier for people to transmit information. The massive amount of videos brings people visual enjoyment, but also carries a lot of redundant information. People want to filter redundant information and quickly find the wonderful shots in the video, which has spawned and promoted the research on the video wonderful shot detection task. The goal of the video wonderful shot detection task is to quickly retrieve the wonderful video clips related to the user query in the corresponding given video when a user query is given. In order to achieve this goal, it is necessary to effectively model the relationship between the video and the query text. However, video and query text are two different forms of information carriers. How to make these two different information carriers fully interact and then efficiently complete the video wonderful shot detection task is a key challenge.

[0003] The current mainstream video moment retrieval method uses deep neural networks to model the interaction between video and query text, and uses the corresponding manually annotated datasets for training. However, this method usually requires extremely high labor costs and it is difficult to obtain sufficient training data. Therefore, some methods use pre-trained image-text models (such as CLIP) to extract features of video frames and text features, and then input these features into a small model and train it with a small amount of manually annotated data, so as to utilize the knowledge of existing mature image-text data to improve the accuracy of video highlight shot detection. Later, large models gradually matured, but most of them were natural language large models, with only decoders but lacking suitable encoders to obtain pre-trained features.

[0004] One of the current prior arts is the patent "A method and system for detecting important shots in videos based on cross-modal reasoning (CN118053096A)". This method first encodes and fuses the visual, audio and text information of the video, and uses the attention mechanism to capture contextual information, and then outputs the importance score of each video clip through a multi-layer perceptron. The cross-modal fusion module of this method can effectively utilize multimodal information and improve the accuracy and comprehensiveness of important shot detection in videos. However, the disadvantage of this method is that it fails to utilize the prior knowledge of existing large models, and there is still room for improvement. In addition, its versatility is poor and it can only be applied to the task of detecting wonderful shots in videos and there are many alternatives. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the existing methods and propose a method and system for detecting wonderful shots with large-scale model-improved feature extraction. The main problem solved by the present invention is how to reduce the dependence on manually annotated data, improve the quality of feature extraction, thereby improving the accuracy and efficiency of video content understanding and retrieval, and improve the generalization of the method.

[0006] In order to solve the above problems, the present invention proposes a wonderful shot detection method for large model improved feature extraction, the method comprising:

[0007] Input a video to be processed, extract video frames at a certain time interval so that the video frames can effectively represent the features within the time period, perform a preprocessing operation on the extracted video frames to obtain preprocessed video frames, and record the number of the preprocessed video frames as N;

[0008] Inputting N preprocessed video frames into the text image language large model BLIP2 to obtain text descriptions of N groups of video frames;

[0009] Input the N pre-processed video frames and the N groups of text descriptions of the video frames into the pre-trained image text model CLIP, extract the overall image features and the overall text features, fuse the overall image features and the overall text features, and obtain the fused feature X∈R N×D , where D is the dimension of a single feature;

[0010] Calculate the shot wonderful score by using the fusion feature X, input the shot wonderful score into the loss function to train the model, and obtain the trained model;

[0011] The user inputs the video and the query statement into the trained model to obtain the wonderful video clips in the video that are related to the query statement.

[0012] Preferably, the preprocessing operation is specifically:

[0013] The pixel size of the video frame is adjusted to 224×224, and the video frame after the pixel size adjustment is normalized so that the RGB value range of each pixel is [0, 1]

[0014] Preferably, the N pre-processed video frames and the N groups of text descriptions of the video frames are input into the pre-trained image text model CLIP, the overall image features and the overall text features are extracted, and the overall image features and the overall text features are fused to obtain the fused feature X∈R N×D , specifically:

[0015] Input the N pre-processed video frames into the pre-trained image text model CLIP, and obtain a one-dimensional vector v for each pre-processed video frame. i ∈R 1×DAs the video image feature of the video frame, i∈[1,N], then for all extracted video frames, the overall image feature V∈R N×D ;

[0016] Input the text descriptions of N groups of video frames into the pre-trained image text model CLIP, and obtain a number of one-dimensional text features for each group of video frame text descriptions, wherein the first one-dimensional text feature is designated as the overall feature t of the group of text descriptions i ∈R 1×D , concatenate the overall features of N groups of text descriptions to get the overall text features, denoted as T∈R N×D ;

[0017] The overall image features and the overall text features are fused according to the following formula:

[0018] X = Norm (V + T, dim = -1),

[0019] Among them, Norm means normalization, and dim=-1 means normalization is performed only on the last dimension, that is, the mean μ and variance σ of the last dimension of the fused feature are calculated, and the mean μ is subtracted from each value of the last dimension of the fused feature and divided by the variance σ.

[0020] Preferably, the method of calculating the shot wonderful score S by using the fusion feature X and inputting the shot wonderful score into the loss function to train the model to obtain the trained model is specifically as follows:

[0021] A linear mapping network Linear is used to map the fusion feature X to dimension 1, and then its value is mapped to [0, 1] through the Sigmoid function to obtain the wonderful scores S∈R of N shots. N×1 , as shown below:

[0022] S = Sigmoid(Linear(X)),

[0023] The score of a shot is s. i , i∈[1,N], the model is trained using the cross entropy loss function using the shot score S, and the true label is denoted by g i , i∈[1,N], then the cross entropy loss function MSE is specifically:

[0024]

[0025] Accordingly, the present invention also provides a wonderful shot detection system for large-model improved feature extraction, comprising:

[0026] The preprocessing unit is used to input the video to be processed, extract video frames at a certain time interval so that the video frames can effectively represent the features within the time period, perform preprocessing operations on the extracted video frames to obtain preprocessed video frames, and record the number of the preprocessed video frames as N; input the N preprocessed video frames into the text image language large model BLIP2 to obtain text descriptions of N groups of video frames;

[0027] A feature extraction and fusion unit is used to input the N pre-processed video frames and the N groups of text descriptions of the video frames into the pre-trained image text model CLIP, extract the overall image features and the overall text features, and fuse the overall image features and the overall text features to obtain a fusion feature X∈R N×D , where D is the dimension of a single feature;

[0028] A shot wonderful score calculation and model training unit, used to calculate the shot wonderful score by using the fusion feature X, input the shot wonderful score into the loss function to train the model, and obtain a trained model;

[0029] The application unit is used for the user to input the video and the query statement into the trained model to obtain the wonderful video clips related to the query statement in the video.

[0030] The implementation of the present invention has the following beneficial effects:

[0031] The video highlight shot detection method based on description enhancement of the present invention can significantly improve the accuracy and efficiency of video content understanding and retrieval. The present invention automatically generates description texts of video frames by utilizing a large natural language processing model, and combines a pre-trained image-text model (such as CLIP) to extract features, which not only reduces the reliance on a large amount of manually annotated data and reduces the cost of the data preparation stage, but also improves the quality and expression ability of features by integrating visual and semantic information. High-quality features enable the model to better understand the video content, thereby enhancing the performance of video highlight shot detection. In addition, the flexibility of the present invention allows it to be easily integrated into existing video analysis frameworks, providing new research directions and technical means for the field of video content analysis, and further promoting the progress of automated video understanding and processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flow chart of a wonderful shot detection method of a large model improved feature extraction according to an embodiment of the present invention;

[0033] Figure 2 It is a structural diagram of a wonderful shot detection system with large model improved feature extraction according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] Figure 1 is a flow chart of a wonderful shot detection method for large model improved feature extraction according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0036] S1, input the video to be processed, extract video frames at a certain time interval so that the video frames can effectively represent the features within the time period, perform preprocessing operations on the extracted video frames to obtain preprocessed video frames, and record the number of preprocessed video frames as N;

[0037] S2, inputting the N pre-processed video frames into the text image language large model BLIP2 to obtain text descriptions of N groups of video frames;

[0038] S3, inputting the N pre-processed video frames and the N groups of text descriptions of the video frames into the pre-trained image text model CLIP, extracting the overall image features and the overall text features, and fusing the overall image features and the overall text features to obtain the fused feature X∈R N×D , where D is the dimension of a single feature;

[0039] S4, using the fusion feature X to calculate the shot wonderful score, and inputting the shot wonderful score into the loss function to train the model to obtain a trained model;

[0040] S5, the user inputs the video and the query statement into the trained model to obtain the wonderful video clips in the video that are related to the query statement.

[0041] Step S1 is as follows:

[0042] S1-1, the time interval for extracting video frames is 2 seconds, and the tool used to extract video frames is ffmpeg tool, which can extract video frames according to time intervals and save them as images;

[0043] S1-2, use torchvision library to adjust the pixel size of the video frame to 224×224, and normalize the video frame after pixel size adjustment so that the RGB value range of each pixel is [0, 1]. Normalization can directly call the Normalize function in the torch framework, and the function parameters are set to mean (0.485, 0.456, 0.406) and variance (0.229, 0.224, 0.225). Both torchvision and torch are machine learning related libraries in python.

[0044] Step S2 is as follows:

[0045] The text-image language model BLIP2 extracts features from video frames and generates text descriptions through its decoding capabilities, converting visual language into natural language;

[0046] The text description usually contains a detailed explanation of the objects, scenes, and possible activities in the image.

[0047] Step S3 is as follows:

[0048] S3-1, inputting N pre-processed video frames into the pre-trained image text model CLIP, and obtaining a one-dimensional vector v for each pre-processed video frame i ∈R 1×D As the video image feature of the video frame, i∈[1,N], then for all extracted video frames, the overall image feature V∈R N×D ;

[0049] S3-2, each group of text descriptions of the video frames includes a number of description texts of different lengths, which are converted into a number of tokens in natural language processing. Therefore, the text descriptions of N groups of video frames are input into the pre-trained image text model CLIP, and a number of one-dimensional text features are obtained, among which the first one-dimensional text feature is designated as the overall feature t of the group of text descriptions. i ∈R 1×D , concatenate the overall features of N groups of text descriptions to get the overall text features, denoted as T∈R N×D , the overall text feature T and the overall image feature V are in one-to-one correspondence;

[0050] S3-3, since the overall text feature and the overall image feature are of the same size and correspond one to one, they can be directly added together and then normalized to complete feature fusion. The overall image feature and the overall text feature are fused according to the following formula:

[0051] X = Norm (V + T, dim = -1),

[0052] Among them, Norm means normalization, and dim=-1 means normalization is performed only on the last dimension, that is, the mean μ and variance σ of the last dimension of the fused feature are calculated, and the mean μ is subtracted from each value of the last dimension of the fused feature and divided by the variance σ.

[0053] Step S4 is as follows:

[0054] S4-1, use a linear mapping network Linear to map the fusion feature X to dimension 1, and then use the Sigmoid function to map its value to [0, 1] to obtain N shot wonderful scores S∈R N×1 , as shown below:

[0055] S = Sigmoid(Linear(X)),

[0056] S4-2, record the wonderful score of a shot as s i , i∈[1,N], the model is trained using the cross entropy loss function using the shot score S, and the true label is denoted by g i , i∈[1,N], then the cross entropy loss function MSE is specifically:

[0057]

[0058] Accordingly, the present invention also provides a wonderful shot detection system for improving feature extraction of a large model, such as Figure 2 As shown, including:

[0059] The preprocessing unit 1 is used to input the video to be processed, extract video frames at a certain time interval so that the video frames can effectively represent the features within the time period, perform preprocessing operations on the extracted video frames to obtain preprocessed video frames, and record the number of the preprocessed video frames as N; input the N preprocessed video frames into the text image language large model BLIP2 to obtain text descriptions of N groups of video frames;

[0060] Specifically, the preprocessing operation includes: adjusting the pixel size of the video frame to 224×224, and normalizing the video frame after the pixel size adjustment so that the RGB value range of each pixel point is [0, 1].

[0061] The feature extraction and fusion unit 2 is used to input the N pre-processed video frames and the N groups of text descriptions of the video frames into the pre-trained image text model CLIP, extract the overall image features and the overall text features, and fuse the overall image features and the overall text features to obtain the fusion feature X∈R N×D , where D is the dimension of a single feature;

[0062] Specifically, N pre-processed video frames are input into the pre-trained image text model CLIP, and for each pre-processed video frame, a one-dimensional vector v is obtained. i ∈R 1×D As the video image feature of the video frame, i∈[1,N], then for all extracted video frames, the overall image feature V∈R N×D ;

[0063] Input the text descriptions of N groups of video frames into the pre-trained image text model CLIP, and obtain a number of one-dimensional text features for each group of video frame text descriptions, wherein the first one-dimensional text feature is designated as the overall feature t of the group of text descriptions i ∈R 1×D , concatenate the overall features of N groups of text descriptions to get the overall text features, denoted as T∈R N×D ;

[0064] The overall image features and the overall text features are fused according to the following formula:

[0065] X = Norm (V + T, dim = -1),

[0066] Among them, Norm means normalization, and dim=-1 means normalization is performed only on the last dimension, that is, the mean μ and variance σ of the last dimension of the fused feature are calculated, and the mean μ is subtracted from each value of the last dimension of the fused feature and divided by the variance σ.

[0067] A shot wonderful score calculation and model training unit 3 is used to calculate the shot wonderful score by using the fusion feature X, input the shot wonderful score into the loss function to train the model, and obtain the trained model;

[0068] Specifically, a linear mapping network Linear is used to map the fusion feature X to dimension 1, and then its value is mapped to [0, 1] through the Sigmoid function to obtain the wonderful scores S∈R of N shots. N×1 , as shown below:

[0069] S = Sigmoid(Linear(X)),

[0070] The score of a shot is s. i , i∈[1,N], the model is trained using the cross entropy loss function using the shot score S, and the true label is denoted by g i , i∈[1,N], then the cross entropy loss function MSE is specifically:

[0071]

[0072] Application unit 4 is used for the user to input the video and query statement into the trained model to obtain the wonderful video clips in the video related to the query statement.

[0073] Therefore, the present invention proposes a method for detecting wonderful shots by improving feature extraction with a large model. This method can significantly improve the accuracy and efficiency of video content understanding and retrieval. This method automatically generates descriptive texts for video frames by utilizing a large natural language processing model, and extracts features in combination with a pre-trained image-text model (such as CLIP). This not only reduces the reliance on a large amount of manually annotated data and reduces the cost of the data preparation stage, but also improves the quality and expressiveness of features by integrating visual and semantic information. High-quality features enable the model to better understand the video content, thereby enhancing the performance of video wonderful shot detection. In addition, the flexibility of this method enables it to be easily integrated into existing video analysis frameworks, providing new research directions and technical means for the field of video content analysis, and further promoting the progress of automated video understanding and processing.

[0074] The above is a detailed introduction to a method and system for detecting wonderful shots with improved feature extraction using a large model provided by an embodiment of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there may be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A wonderful shot detection method for large model improved feature extraction, characterized in that: The method comprises: Input a video to be processed, extract video frames at a certain time interval so that the video frames can effectively represent the features within the time period, perform a preprocessing operation on the extracted video frames to obtain preprocessed video frames, and record the number of the preprocessed video frames as N; Inputting N preprocessed video frames into the text image language large model BLIP2 to obtain text descriptions of N groups of video frames; Input the N pre-processed video frames and the N groups of text descriptions of the video frames into the pre-trained image text model CLIP, extract the overall image features and the overall text features, fuse the overall image features and the overall text features, and obtain the fused feature X∈R N×D , where D is the dimension of a single feature; Calculate the shot wonderful score by using the fusion feature X, input the shot wonderful score into the loss function to train the model, and obtain the trained model; The user inputs the video and the query statement into the trained model to obtain the wonderful video clips in the video that are related to the query statement.

2. The method for detecting wonderful shots by improving feature extraction using a large model as claimed in claim 1, characterized in that: The pre-processing operation is specifically as follows: The pixel size of the video frame is adjusted to 224×224, and the video frame after the pixel size adjustment is normalized so that the RGB value range of each pixel is [0, 1].

3. The method for detecting wonderful shots by improving feature extraction using a large model as claimed in claim 1, characterized in that: The N pre-processed video frames and the N groups of text descriptions of the video frames are input into the pre-trained image text model CLIP, the overall image features and the overall text features are extracted, and the overall image features and the overall text features are fused to obtain the fused feature X∈R N×D , specifically: Input the N pre-processed video frames into the pre-trained image text model CLIP, and obtain a one-dimensional vector v for each pre-processed video frame. i ∈R 1×D As the video image feature of the video frame, i∈[1,N], then for all extracted video frames, the overall image feature V∈R N×D ; Input the text descriptions of N groups of video frames into the pre-trained image text model CLIP, and obtain a number of one-dimensional text features for each group of video frame text descriptions, wherein the first one-dimensional text feature is designated as the overall feature t of the group of text descriptions i ∈R 1×D , concatenate the overall features of N groups of text descriptions to get the overall text features, denoted as T∈R N×D ; The overall image features and the overall text features are fused according to the following formula: X=Norm(V+T,dim=-1), Among them, Norm means normalization, and dim=-1 means normalization is performed only on the last dimension, that is, the mean μ and variance σ of the last dimension of the fused feature are calculated, and the mean μ is subtracted from each value of the last dimension of the fused feature and divided by the variance σ.

4. The method for detecting wonderful shots by improving feature extraction using a large model as claimed in claim 1, characterized in that: The fusion feature X is used to calculate the shot wonderful score S, and the shot wonderful score is input into the loss function to train the model to obtain the trained model, specifically: A linear mapping network Linear is used to map the fusion feature X to dimension 1, and then its value is mapped to [0,1] through the Sigmoid function to obtain the wonderful scores S∈R of N shots. N×1 , as shown below: S = Sigmoid(Linear(X)), The score of a shot is s. i , i∈[1,N], the model is trained using the cross entropy loss function using the shot score S, and the true label is denoted by g i , i∈[1,N], then the cross entropy loss function MSE is specifically:

5. A wonderful shot detection system with improved feature extraction of a large model, characterized in that: The system comprises: The preprocessing unit is used to input the video to be processed, extract video frames at a certain time interval so that the video frames can effectively represent the features within the time period, perform preprocessing operations on the extracted video frames to obtain preprocessed video frames, and record the number of the preprocessed video frames as N; input the N preprocessed video frames into the text image language large model BLIP2 to obtain text descriptions of N groups of video frames; A feature extraction and fusion unit is used to input the N pre-processed video frames and the N groups of text descriptions of the video frames into the pre-trained image text model CLIP, extract the overall image features and the overall text features, and fuse the overall image features and the overall text features to obtain a fusion feature X∈R N×D , where D is the dimension of a single feature; A shot wonderful score calculation and model training unit, used to calculate the shot wonderful score by using the fusion feature X, input the shot wonderful score into the loss function to train the model, and obtain a trained model; The application unit is used for the user to input the video and the query statement into the trained model to obtain the wonderful video clips related to the query statement in the video.

6. A wonderful shot detection system with improved feature extraction using a large model as claimed in claim 5, characterized in that: The preprocessing operations in the preprocessing unit are specifically: The pixel size of the video frame is adjusted to 224×224, and the video frame after the pixel size adjustment is normalized so that the RGB value range of each pixel is [0, 1].

7. A wonderful shot detection system with improved feature extraction using a large model as claimed in claim 5, characterized in that: The N pre-processed video frames and the N groups of text descriptions of the video frames are input into the pre-trained image text model CLIP, the overall image features and the overall text features are extracted, and the overall image features and the overall text features are fused to obtain the fused feature X∈R N×D , specifically: Input the N pre-processed video frames into the pre-trained image text model CLIP, and obtain a one-dimensional vector v for each pre-processed video frame. i ∈R 1×D As the video image feature of the video frame, i∈[1,N], then for all extracted video frames, the overall image feature V∈R N×D ; Input the text descriptions of N groups of video frames into the pre-trained image text model CLIP, and obtain a number of one-dimensional text features for each group of video frame text descriptions, wherein the first one-dimensional text feature is designated as the overall feature t of the group of text descriptions i ∈R 1×D , concatenate the overall features of N groups of text descriptions to get the overall text features, denoted as T∈R N×D ; The overall image features and the overall text features are fused according to the following formula: X=Norm(V+T,dim=-1), Among them, Norm means normalization, and dim=-1 means normalization is performed only on the last dimension, that is, the mean μ and variance σ of the last dimension of the fused feature are calculated, and the mean μ is subtracted from each value of the last dimension of the fused feature and divided by the variance σ.

8. A wonderful shot detection system with large model improved feature extraction as claimed in claim 5, characterized in that: The fusion feature X is used to calculate the shot wonderful score S, and the shot wonderful score is input into the loss function to train the model to obtain the trained model, specifically: A linear mapping network Linear is used to map the fusion feature X to dimension 1, and then its value is mapped to [0,1] through the Sigmoid function to obtain the wonderful scores S∈R of N shots. N×1 , as shown below: S = Sigmoid(Linear(X)), The score of a shot is s. i , i∈[1,N], the model is trained using the cross entropy loss function using the shot score S, and the true label is denoted by g i , i∈[1,N], then the cross entropy loss function MSE is specifically:

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images