Video text retrieval method based on compressed domain video stream

Abstract

The invention discloses a video text retrieval method based on a compressed domain video stream, relates to the technical field of video processing, and solves the technical problem that when a model is trained, if a part of time sequence information is lost, the retrieval capability of the trained model is possibly insufficient. The method comprises the following steps: acquiring a training data set of videos and texts, and preprocessing the videos in the training data set; extracting key frame features of videos in the data set by using an image encoder, and integrating the key frame features in an average pooling mode to obtain video features; constructing a compression feature extraction encoder by using a convolutional layer and a lightweight visual converter, and extracting video features; constructing a compressed video feature fusion network, and predicting the key features to obtain predicted video features; generating word features and sentence features by using a text encoder; calculating the matching degree of the video and the text to obtain a final matching result; and the retrieval efficiency and accuracy can be improved.

Description

Technical Field

[0001] The invention belongs to the field of video processing, relates to a video text retrieval technology, and specifically is a video text retrieval method based on a compressed domain video stream. Background Art

[0002] Video text retrieval refers to querying semantically related video content through given text; or retrieving relevant semantic descriptions through video content; it involves feature representation extraction, feature embedding and matching of video and text, as well as optimization of objective functions; with the increasing number of videos uploaded to the Internet every day, video text retrieval has become a new requirement for people to efficiently find relevant videos; therefore, deep learning technology has made significant progress in this field, especially the emergence of large-scale image-text pre-training models, which has achieved strong performance on the benchmark of video text retrieval.

[0003] In the prior art, videos are regarded as image sequences, and uniform frame extraction technology is used to capture video content. Part of the timing information may be lost during the processing. This timing information includes the movement and contour changes of objects in the video, and is an important factor in constructing a video. When training a model, if part of the timing information is lost, the retrieval capability of the trained model may be insufficient, and the retrieved video or text may not meet the user's expectations.

[0004] The present invention provides a video text retrieval method based on compressed domain video stream to solve the above technical problems. Summary of the invention

[0005] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a video text retrieval method based on a compressed domain video stream, which is used to solve the technical problem that when training a model, if a part of the timing information is lost, the retrieval ability of the trained model may be insufficient, and the retrieved video or text may not meet the user's expectations.

[0006] To achieve the above object, a first aspect of the present invention provides a video text retrieval method based on a compressed domain video stream, comprising:

[0007] Step S1: obtaining a training data set of videos and texts, and preprocessing the videos in the training data set;

[0008] Step S2: extract key frame features of the video in the dataset using an image encoder, integrate the key frame features by average pooling, and obtain video features;

[0009] Step S3: construct a compression feature extraction encoder using a convolutional layer and a lightweight visual converter, and extract video features using the compression feature extraction encoder;

[0010] Step S4: construct a compressed video feature fusion network, and use the compressed video feature fusion network to predict key features to obtain predicted video features.

[0011] Preferably, the method further includes: step S5: using a text encoder in a large-scale image-text pre-training model to generate word features and sentence features; calculating the matching degree of several videos and texts to obtain a final matching result.

[0012] Preferably, calculating the matching degree of the plurality of videos and texts to obtain the final matching result includes:

[0013] Retrieving video features, key frame features, predicted frame features, predicted video features, word features, and sentence features; performing matrix or vector multiplication of the video features, key frame features, predicted frame features, predicted video features, word features, and sentence features respectively;

[0014] The weights are obtained through Softmax and summed to obtain the similarities in different dimensions; the average values ​​of the similarities in different dimensions are calculated and the average value is marked as the final matching degree; the videos or texts are sorted from large to small according to the final matching degree to obtain the most similar videos or texts.

[0015] The present invention calculates the matching degree between video and text from multiple dimensions, and obtains the most similar video or text according to the average value of the similarity in different dimensions; it can analyze the similarity between video and text from multiple angles, which is conducive to improving the accuracy of retrieval results.

[0016] Preferably, the preprocessing of the videos in the training data set includes:

[0017] Retrieve the video in the training data set and encode it using traditional video encoding to obtain a video v; each video consists of N groups of pictures, each picture contains a key frame Ii and several prediction frames i represents the i-th group of pictures,

[0018] Perform semi-decoding on the first predicted frame and extract motion vectors and residual data Among them, H and W represent the height and width of the feature respectively; mv represents the motion vector, res represents the residual, and R represents a real number set.

[0019] It should be noted that i=1, 2, ..., n, where n is a positive integer; the number of each group of pictures is set to X; and the number of each group of pictures is greater than or equal to 2; traditional video encoding includes MPEG-4, H.264, H.265, etc.; represents the first motion vector of the i-th group of pictures; Represents the first residual of the i-th group of pictures.

[0020] The present invention performs semi-decoding processing on the first predicted frame of the video in the training data set, which can screen out impossible results in the early stage of processing, thereby reducing subsequent unnecessary calculations; the number of candidates that need to be evaluated is reduced, and the semi-decoding processing is conducive to accelerating the decoding process.

[0021] Preferably, the step of integrating key frame features by average pooling to obtain video features includes:

[0022] The video is sampled at a sampling rate of Z per second, and the image encoder of the large-scale image-text pre-training model is used to extract the feature representation of the key frame Ii

[0023] By formula The features of all sampled key frames are averaged to obtain the video features v of the entire video f .

[0024] The present invention can effectively reduce the dimension of the feature vector by performing average pooling on the key frame features; it is helpful to reduce the computational burden and the risk of overfitting; compared with maximum pooling, average pooling can better preserve the spatial information of the feature vector to a certain extent; and it lays the foundation for subsequent analysis.

[0025] Preferably, the extracting of video features by using a compression feature extraction encoder comprises:

[0026] Calling a compression feature extraction encoder; wherein the compression feature extraction encoder is composed of L ViT encoders stacked together, each encoder including: a normalization layer, a multi-head attention mechanism layer, and a multi-layer perception layer;

[0027] The number of channels of the motion vector feature convolution layer is set to m; the number of channels of the residual feature convolution layer is set to r;

[0028] By formula Calculate the motion vector features of the first predicted frame in the i-th group of pictures;

[0029] By formula Calculate the residual features of the first predicted frame in the i-th group; where Transformer-Enc is a compressed feature extraction encoder and conv represents a convolutional layer.

[0030] Preferably, the method of predicting key features using a compressed video feature fusion network to obtain predicted video features includes:

[0031] Calling a compressed video feature fusion network; wherein the compressed video feature fusion network includes a channel weight predictor, a spatial attention module and a channel attention module;

[0032] The motion vector is upsampled by bilinear interpolation method, and the formula is expressed as ; Where Interpolate represents the upsampling operation, mv is the incoming motion vector; f mv is the input motion vector feature; The spatial scale that represents the motion vector characteristics;

[0033] The channel weight prediction values ​​of motion vector and residual are calculated through the channel attention module and weight predictor, and the spatial weight prediction values ​​of motion vector and residual are extracted through the spatial attention module and activation function;

[0034] The key frame features are updated using the weighted prediction values ​​of the motion vector and the weighted prediction values ​​of the residual to obtain the predicted frame features. The formula is:

[0035]

[0036] Among them, the weight prediction value includes the channel weight prediction value and the space weight prediction value; is matrix multiplication; channel_weight mv Indicates the channel weight prediction value of the motion vector; spatial_weight mv Indicates the spatial weight prediction value of the motion vector; channel_weight res Represents the channel weight prediction value of the residual; spatial_weight res Represents the spatial weighted predicted value of the residual;

[0037] By formula The predicted frame features are averaged to obtain the predicted video features.

[0038] Preferably, the channel weight prediction value of the motion vector and the residual is calculated by the channel attention module and the weight predictor, comprising:

[0039] By formula weight ch =Concat(ReLU(Conv(Cancat(p mv ,mv,f mv ),W)+b)) calculate the channel weight of the motion vector;

[0040] The channel weight of the motion vector is passed through the channel weight predictor, and the formula channel_weight mv=Sigmoid(FC(ReLU(FC(AvgPool(weight ch ))))))Calculate the channel weight prediction value of the motion vector;

[0041] Among them, Concat represents tensor concatenation, W represents the weight of the convolution kernel, and b represents bias; AvgPool represents average pooling, FC represents full connection, Sigmoid and ReLU represent activation functions;

[0042] The channel weight prediction value channel_weight of the residual is calculated through the above operation and formula res .

[0043] Preferably, the extracting of the spatial weight prediction value of the motion vector and the residual by the spatial attention module and the activation function includes:

[0044] By formula weight spa =Concat(ReLU(Conv(Concat(p mv ,mv,f mv ),W)+b)) calculates the spatial weight of the motion vector; through the formula spatial_weight mv =Softmax(weight spa ) Calculate the spatial weight prediction value of the motion vector;

[0045] Among them, Concat represents tensor concatenation, ReLU represents activation function, W represents the weight of convolution kernel, b represents bias, Softmax represents normalization,

[0046] The spatial weight of the residual is calculated by the above operation and formula spatial_weight res .

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] 1. The present invention performs semi-decoding processing on the first predicted frame of the video in the training data set, which can screen out impossible results in the early stage of processing, thereby reducing subsequent unnecessary calculations; the number of candidates that need to be evaluated is reduced, and the semi-decoding processing is conducive to accelerating the decoding process; by averaging the key frame features, the dimension of the feature vector can be effectively reduced; it is conducive to reducing the computational burden and the risk of overfitting; compared with maximum pooling, average pooling better retains the spatial information of the feature vector to a certain extent; and lays the foundation for subsequent analysis.

[0049] 2. The present invention directly performs video text retrieval in the compressed domain; directly uses compressed video for video text retrieval, effectively reducing the processing time and computing resources required for decompressing video data in traditional retrieval; constructs a more accurate and robust video representation by combining multi-level comparison and compressed domain information; multi-level comparison enables the system to carefully analyze the correlation between video content and query text, and compressed domain information can avoid a large amount of time series information loss, thereby improving the accuracy of retrieval; a lightweight visual transformer (ViT) is used to extract compressed domain information features, and spatial channel fusion technology is used to generate predicted frame features, making full use of compressed domain features while maintaining consistency with key frames. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0051] Figure 1 It is a schematic diagram of the overall steps of the present invention;

[0052] Figure 2 It is a schematic diagram of the step structure of the present invention. DETAILED DESCRIPTION

[0053] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] See also Figure 1-Figure 2 The first aspect of the present invention provides a video text retrieval method based on a compressed domain video stream, comprising:

[0055] Step S1: obtaining a training data set of videos and texts, and preprocessing the videos in the training data set;

[0056] Step S2: extract key frame features of the video in the dataset using an image encoder, integrate the key frame features by average pooling, and obtain video features;

[0057] Step S3: construct a compression feature extraction encoder using a convolutional layer and a lightweight visual converter, and extract video features using the compression feature extraction encoder;

[0058] Step S4: construct a compressed video feature fusion network, and use the compressed video feature fusion network to predict key features to obtain predicted video features;

[0059] Step S5: Generate word features and sentence features using the text encoder in the large-scale image-text pre-training model; calculate the matching degree of several videos and texts to obtain the final matching result.

[0060] Obtain a training data set of videos and texts, encode the videos using traditional video encoding methods, and obtain videos v; each video consists of N groups of pictures, each picture contains a key frame I i and several predicted frames i represents the i-th group of pictures, Perform semi-decoding on the first predicted frame and extract motion vectors and residual data Among them, H and W represent the height and width of the feature respectively.

[0061] It should be noted that i=1, 2, ..., n, where n is a positive integer; the number of each group of pictures is set to X; and the number of each group of pictures is greater than or equal to 2; traditional video encoding includes MPEG-4, H.264, H.265, etc.

[0062] The key frame features of the videos in the dataset are extracted using an image encoder. The videos are sampled at a sampling rate of Z per second. The key frame I is extracted using the image encoder of the large-scale image-text pre-training model. i The feature representation By formula The features of all sampled key frames are averaged to obtain the video features v of the entire video f .

[0063] It should be noted that Z is greater than or equal to 3, and the large-scale image-text pre-training model includes an image encoder and a text encoder, which process video images and texts respectively.

[0064] Call the compression feature extraction encoder; the compression feature extraction encoder is composed of L ViT encoders stacked, each of which includes: a normalization layer, a multi-head attention mechanism layer, and a multi-layer perception layer; set the number of channels of the motion vector feature convolution layer to m; set the number of channels of the residual feature convolution layer to r; through the formula Calculate the motion vector features of the first predicted frame in the i-th group of pictures; Transformer-Enc is a compression feature extraction encoder, Conv represents a convolutional layer; through the formula Calculate the residual features of the first predicted frame in the i-th group; where Transformer-Enc is a compressed feature extraction encoder and conv represents a convolutional layer.

[0065] The compressed video feature fusion network is called; the compressed video feature fusion network includes a channel weight predictor, a spatial attention module, and a channel attention module; the motion vector is upsampled by a bilinear interpolation method, and the formula is expressed as ; Where Interpolate represents the upsampling operation, mv is the incoming motion vector; f mv is the input motion vector feature; The spatial scale that represents the motion vector characteristics;

[0066] It should be noted that the interpolation method uses bilinear interpolation, which is a method of performing linear interpolation in two directions and is often used for image enlargement. In the present invention, it is used to adjust the size of the motion vector to be the same as the motion vector feature for subsequent further processing.

[0067] By formula weight ch =Concat(Re LU(Conv(Concat(p mv ,mv,f mv ),W)+b)) calculate the channel weight of the motion vector;

[0068] The channel weight of the motion vector is passed through the channel weight predictor, and the formula channel_weight mv =Sigmoid(FC(Re LU(FC(AvgPool(weight ch ))))))Calculate the channel weight prediction value of the motion vector;

[0069] Among them, Concat represents tensor concatenation, W represents the weight of the convolution kernel, and b represents the bias; AvgPool represents average pooling, FC represents full connection, Sigmoid and ReLU represent activation functions; the channel weight prediction value channel_weight of the residual is calculated through the above operations and formulas res .

[0070] By formula weight spa =Concat(ReLU(Conv(Concat(p mv ,mv,f mv ),W)+b)) calculates the spatial weight of the motion vector; through the formula spatial_weight mv =Softmax(weight spa) calculates the spatial weight prediction value of the motion vector; where Concat represents tensor concatenation, Re LU represents activation function, W represents the weight of the convolution kernel, b represents bias, and Softmax represents normalization; the spatial weight spatial_weight of the residual is calculated through the above operation and formula res .

[0071] It should be noted that the channel weight predictor consists of multiple linear layers and activation functions, which can accurately predict the importance of each channel; tensor concatenation merges multiple tensors along the specified dimension into a larger tensor, which is used to transfer and fuse information between feature maps of different layers, so that these features can be considered simultaneously in subsequent layers, which helps the model capture richer information.

[0072] It should be noted that the use of the Softmax function for normalization is conducive to more effective weight adjustment of the model during the training process; its function is to compress a K-dimensional vector containing arbitrary real numbers into a vector of length K, in which the value of each element is between 0 and 1, and the sum of all elements is 1, forming a concept distribution; the output of the Softmax function changes smoothly, which is conducive to model training.

[0073] The key frame features are updated using the weighted prediction values ​​of the motion vector and the weighted prediction values ​​of the residual to obtain the predicted frame features. The formula is:

[0074]

[0075] Among them, the weight prediction value includes the channel weight prediction value and the space weight prediction value; is matrix multiplication; channel_weight mv Indicates the channel weight prediction value of the motion vector; spatial_weight mv Indicates the spatial weight prediction value of the motion vector; channel_weight res Represents the channel weight prediction value of the residual; spatial_weight res Represents the spatial weight prediction value of the residual; through the formula The predicted frame features are averaged to obtain the predicted video features.

[0076] Retrieving video features, key frame features, predicted frame features, predicted video features, word features, and sentence features; performing matrix or vector multiplication of the video features, key frame features, predicted frame features, predicted video features, word features, and sentence features respectively;

[0077] The weights are obtained through Softmax and summed to obtain the similarities in different dimensions; the average values ​​of the similarities in different dimensions are calculated and the average value is marked as the final matching degree; the videos or texts are sorted from large to small according to the final matching degree to obtain the most similar videos or texts.

[0078] It should be noted that during the training process, given a batch of B video-text pairs, the model will generate a B×B similarity matrix and use the differences on the symmetric similarity matrix to optimize the retrieval model.

[0079] Part of the data in the above formula is calculated by removing the dimension and taking its numerical value. The formula is a formula closest to the actual situation obtained by software simulation of a large amount of collected data; the preset parameters and preset thresholds in the formula are set by technical personnel in this field according to actual conditions or obtained through simulation of a large amount of data.

[0080] The working principle of the present invention is as follows: the present invention obtains a training data set of videos and texts, and pre-processes the videos in the training data set; uses an image encoder to extract key frame features of the videos in the data set, integrates the key frame features by average pooling, and obtains video features; uses a convolutional layer and a lightweight visual converter to construct a compressed feature extraction encoder, and uses the compressed feature extraction encoder to extract video features; constructs a compressed video feature fusion network, and uses the compressed video feature fusion network to predict key features to obtain predicted video features; uses a text encoder in a large-scale image text pre-training model to generate word features and sentence features; calculates the matching degree of several videos and texts to obtain a final matching result.

[0081] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A video text retrieval method based on compressed domain video stream, characterized in that: include: Step S1: obtaining a training data set of videos and texts, and preprocessing the videos in the training data set; Step S2: extract key frame features of the video in the dataset using an image encoder, integrate the key frame features by average pooling, and obtain video features; Step S3: construct a compression feature extraction encoder using a convolutional layer and a lightweight visual converter, and extract video features using the compression feature extraction encoder; Step S4: construct a compressed video feature fusion network, and use the compressed video feature fusion network to predict key features to obtain predicted video features.

2. The video text retrieval method based on compressed domain video stream according to claim 1, characterized in that: It also includes: Step S5: using the text encoder in the large-scale image-text pre-training model to generate word features and sentence features; calculating the matching degree between several videos and texts to obtain the final matching result.

3. The video text retrieval method based on compressed domain video stream according to claim 2 is characterized in that: The calculation of the matching degree of the plurality of videos and the texts to obtain the final matching result includes: Retrieving video features, key frame features, predicted frame features, predicted video features, word features, and sentence features; performing matrix or vector multiplication of the video features, key frame features, predicted frame features, predicted video features, word features, and sentence features respectively; The weights are obtained through Softmax and summed to obtain the similarities in different dimensions; the average values ​​of the similarities in different dimensions are calculated and the average value is marked as the final matching degree; the videos or texts are sorted from large to small according to the final matching degree to obtain the most similar videos or texts.

4. The video text retrieval method based on compressed domain video stream according to claim 1, characterized in that: The preprocessing of the videos in the training data set includes: Retrieve the video in the training data set and encode it using traditional video encoding to obtain a video v; each video consists of N groups of pictures, each picture contains a key frame Ii and several prediction frames i represents the i-th group of pictures, Perform semi-decoding on the first predicted frame and extract motion vectors and residual data Among them, H and W represent the height and width of the feature respectively; mv represents the motion vector, res represents the residual, and R represents a real number set.

5. The video text retrieval method based on compressed domain video stream according to claim 1 is characterized in that: The key frame features are integrated by average pooling to obtain video features, including: The video is sampled at a sampling rate of Z per second, and the image encoder of the large-scale image-text pre-training model is used to extract the feature representation of the key frame Ii By formula The features of all sampled key frames are averaged to obtain the video features v of the entire video f .

6. The video text retrieval method based on compressed domain video stream according to claim 1, characterized in that: The method of extracting video features by using a compression feature extraction encoder includes: Calling a compression feature extraction encoder; wherein the compression feature extraction encoder is composed of L ViT encoders stacked together, each of which includes: a normalization layer, a multi-head attention mechanism layer, and a multi-layer perception layer; The number of channels of the motion vector feature convolution layer is set to m; the number of channels of the residual feature convolution layer is set to r; By formula Calculate the motion vector features of the first predicted frame in the i-th group of pictures; By formula Calculate the residual features of the first predicted frame in the i-th group; where Transformer-Enc is a compressed feature extraction encoder and conv represents a convolutional layer.

7. The video text retrieval method based on compressed domain video stream according to claim 1 is characterized in that: The method of using the compressed video feature fusion network to predict key features to obtain predicted video features includes: Calling a compressed video feature fusion network; wherein the compressed video feature fusion network includes a channel weight predictor, a spatial attention module and a channel attention module; The motion vector is upsampled by bilinear interpolation method, and the formula is expressed as p mv = Among them, Interpolate represents the upsampling operation, mv is the input motion vector; f mv is the input motion vector feature; The spatial scale that represents the motion vector characteristics; The channel weight prediction values ​​of motion vector and residual are calculated through the channel attention module and weight predictor, and the spatial weight prediction values ​​of motion vector and residual are extracted through the spatial attention module and activation function; The key frame features are updated using the weighted prediction values ​​of the motion vector and the weighted prediction values ​​of the residual to obtain the predicted frame features f i P ; The formula is: Among them, the weight prediction value includes the channel weight prediction value and the space weight prediction value; is matrix multiplication; channel_weight mv Indicates the channel weight prediction value of the motion vector; spatial_weight mv Indicates the spatial weight prediction value of the motion vector; channel_wehght res Represents the channel weight prediction value of the residual; spatial_weight res Represents the spatial weighted predicted value of the residual; By formula The predicted frame features are averaged to obtain the predicted video features.

8. The video text retrieval method based on compressed domain video stream according to claim 7, characterized in that: The channel weight prediction value of the motion vector and the residual is calculated by the channel attention module and the weight predictor, including: By formula weight ch =Concat(Re LU(Conv(Concat(p mv ,mv,f mv ),W)+b)) calculate the channel weight of the motion vector; The channel weight of the motion vector is passed through the channel weight predictor, and the formula channel_weight mv =Sigmoid(FC(ReLU(FC(AvgPool(weight ch )))))Calculate the channel weight prediction value of the motion vector; Among them, Concat represents tensor concatenation, W represents the weight of the convolution kernel, and b represents bias; AvgPool represents average pooling, FC represents full connection, Sigmoid and ReLU represent activation functions; The channel weight prediction value channel_weight of the residual is calculated through the above operation and formula res .

9. The video text retrieval method based on compressed domain video stream according to claim 7, characterized in that: The method of extracting the spatial weight prediction value of the motion vector and the residual through the spatial attention module and the activation function includes: By formula weight spa =Concat(ReLU(Conv(Concat(pmv,mv,f mv ),W)+b)) calculates the spatial weight of the motion vector; through the formula spatial_weight mv =Softmax(weight spa ) Calculate the spatial weight prediction value of the motion vector; Among them, Concat represents tensor concatenation, Re LU represents activation function, W represents the weight of convolution kernel, b represents bias, Softmax represents normalization, The spatial weight of the residual is calculated by the above operation and formula spatial_wight res .

Images