A homologous video retrieval method and system based on temporal features

Abstract

The present invention discloses a homologous video retrieval method and system based on temporal features, including extracting key frames from videos, using a deep residual convolutional neural network to extract features from key frames, using two submodules to extract corresponding frame temporal feature information and frame importance information, obtaining an overall feature vector representation of the video after feature aggregation, using a twin network to construct positive and negative samples to train network parameters, and after obtaining a network model with fixed parameters, extracting features from the protected video and the retrieved video respectively and detecting whether the similarity exceeds a threshold, and returning the detection result. The deep residual convolutional neural network provided by the present invention realizes feature extraction, and the two submodules of temporal feature information extraction and frame importance information extraction realize the detection of interference frames and clipping and order swapping of infringing videos, thereby improving the effect of detection coverage of multiple infringement methods and the accuracy of homologous video retrieval.

Description

Technical Field

[0001] The present invention belongs to the technical field of video retrieval, and in particular relates to a homologous video retrieval method and system based on temporal features. Background Art

[0002] In the past, with the continuous development of information technology, different forms of digital works circulated on the Internet. Due to the rise of major short video platforms, short videos have become a more widely used medium among different forms of digital works. People share and record their lives and creative designs by publishing short videos. However, due to the easy theft and tampering of digital content on the Internet, infringers can often simply change other people's works or content by adding watermarks, changing frame rates, changing tones, adjusting the screen ratio, etc., and then steal them. Some thieves even use a combination of these operations, making infringement detection in the video field a difficult problem.

[0003] People have begun to explore different detection methods to try to protect the intellectual copyright of digital works such as videos. At present, infringement detection in the video field generally requires a near duplicate video retrieval method (NDVR). The goal of NDVR is to compare the similarity of the video to be checked for infringement (query video) with the query video one by one in the video library to be retrieved that may have infringed the copyright, and return the video that exceeds a certain threshold as a suspected infringement result.

[0004] After the rise of deep learning technology, existing homology video retrieval methods generally extract key frames from videos. The neural network parameters are trained using labeled training data until convergence. During retrieval, the trained convolutional neural network is used to extract features from key frames, and then different comparison methods are used to obtain the similarity between video pairs. Different comparison methods can be roughly divided into:

[0005] (1) The feature vectors of all key frames are aggregated by averaging or other methods to obtain the overall feature vector representation of the video, and the cosine distance between the video feature vectors is calculated to obtain the similarity.

[0006] (2) The distance between the key frames of the video and the key frame feature vectors is calculated as the similarity between frames, and the sequence relationship of each key frame in the video is analyzed using methods such as Bag-of-Words (BoW).

[0007] However, both of the above methods have some defects in some common infringement scenarios. Existing algorithms often evenly extract a certain number of frames as key frames in each time-length segment of the video, and fail to effectively consider the timing relationship information between frames. However, infringers often insert interference frames into the original video, add large-area occlusions or watermark text and other content to the key frames of the original video, and then intercept the original video and sew the segments into their own infringing videos, which has a certain performance impact on the existing detection methods. In addition, the existing neural network training method makes the model parameters relatively fixed, and the performance will decline when faced with a large number of complex and changeable infringement methods, which makes the performance of detecting homologous videos limited. Summary of the invention

[0008] In view of the deficiencies in the prior art, the present invention provides a method and system for retrieving homologous videos based on temporal features.

[0009] To achieve the above purpose, the technical solution provided by the present invention is:

[0010] The present invention discloses a homologous video retrieval method based on temporal features, which comprises the following steps:

[0011] (1) Build a neural network model and use open source data sets to train the network model parameters until the model parameters converge and then fix the network model parameters;

[0012] (2) Use the trained model to extract the corresponding feature vectors for the videos that need to be protected and retrieved;

[0013] The step (1) specifically comprises the following steps:

[0014] (1-1) Key frame extraction: query and retrieve the video into frames, extract a certain number of frames within a certain duration according to the video length, and finally sample the extracted frames and scale them to a specific size to obtain the video key frames;

[0015] (1-2) Frame feature extraction: The video key frames are input into the deep residual convolutional neural network pre-trained on the ImageNet dataset, and forward propagation is performed to extract features from each channel of the frame image. After pooling, activation, and regularization operations, the vector output by the last average pooling layer is extracted as the feature vector of each frame;

[0016] (1-3) Temporal feature extraction: The feature vectors of each frame are input into the Transformer encoder in the temporal order in which the frames appear in the video, and the temporal information feature vectors between frames are obtained;

[0017] (1-4) Frame importance information extraction: Use the attention mechanism to extract the mapping of different semantic spaces from the feature vectors of each frame, obtain the feature vector containing the importance information of each frame, and obtain the frame importance information feature vector after splicing and aggregation;

[0018] (1-5) Feature aggregation: The output feature vectors of the self-attention mechanism and the temporal feature extraction module are input into the fully connected layer for weighted aggregation to obtain the overall feature vector representation of the video;

[0019] (1-6) Twin network training: When training network parameters, in each training round, for each input video, a video with the same source as the video is extracted as a positive sample, and a video with a different source from the video is extracted as a negative sample. The three samples are constructed into a triplet, and the loss value is obtained using the triplet loss function. The gradient is calculated using the loss value, and then the neural network is fed back and propagated to update the parameters of each layer. This step is repeated until the network parameters converge.

[0020] Step (2) specifically includes the following steps:

[0021] (2-1) Constructing a video database that needs to be protected: The user-generated videos that the system needs to protect are stored in the database. The feature vector of each protected video is obtained through step (1). The feature vectors of all videos are stored in the database to obtain the feature matrix of all protected videos.

[0022] (2-2) Extracting features from the retrieved video: For videos that need to be detected to see if they infringe the rights of the videos in the protected video library, extract the corresponding video feature vector through step (1) and perform a comparative search.

[0023] Step (3) specifically includes the following steps:

[0024] (3-1) Setting a similarity search threshold: Calculate the similarity between the searched video and the protected video. The similarity is in the range of (0,1), where 1 means that the model determines that the searched video and the protected video are 100% similar, 0 means that they are completely dissimilar, and other values ​​represent the percentage of similarity. If a search threshold is set, the system will return video pairs with similarities exceeding the threshold as suspected infringement results for the user to make further judgments later.

[0025] (3-2) Calculate similarity: The feature vectors of the searched video obtained in step (2) are multiplied one by one with the feature matrix in the protected video library to obtain the cosine distance between the feature vectors of the searched video and each protected video as the similarity; the similarities are compared one by one to determine whether they exceed the set threshold. If so, the video pair is returned as the system detection result; if not, the suspected infringing video is not detected as the system detection result. ) Set the similarity search threshold and perform similarity calculation.

[0026] A homologous video retrieval based on temporal features, comprising:

[0027] The model building and training module is used to build a neural network model and train the network model parameters until the model parameters converge and then fix the network model parameters;

[0028] A feature extraction module extracts features and forms a protection video feature matrix and stores it in a database;

[0029] The similarity calculation module is used to compare and retrieve videos to see if they infringe copyright.

[0030] Beneficial effects of the present invention:

[0031] The method of the present invention extracts key frames from the video, uses the deep residual convolutional neural network ResNet with fixed parameters pre-trained on the large-scale dataset ImageNet to extract feature vectors from each frame, and uses the self-attention mechanism and Transformer encoder to extract and parse the importance and temporal information of each frame. After operations such as aggregation and weighting, the feature vector representation of the video is obtained, and the metric learning method is used to train the model parameters, so that the feature vector expressions of the same source video are closer in the semantic space, while the feature vectors of different source videos are farther away in the semantic space. The cosine distance between the feature vectors of different videos is calculated to compare and measure the similarity between the videos, and the video results suspected of infringing are retrieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a schematic diagram of a homologous video retrieval method and system flow based on temporal features;

[0033] Figure 2 Schematic diagram of the process of extracting frame features for deep residual convolutional neural network;

[0034] Figure 3 It is a schematic diagram of the temporal feature extraction, frame importance information extraction and feature aggregation module;

[0035] Figure 4 Schematic diagram of the feature extraction and similarity calculation module flow. DETAILED DESCRIPTION

[0036] In order to further understand the content of the present invention, the present invention is described in detail in conjunction with examples. The following examples are used to illustrate the present invention but are not used to limit the scope of the present invention.

[0037] Example 1

[0038] like Figure 1As shown, this embodiment relates to a homologous video retrieval system based on time series features, which includes a model construction and training module, a feature extraction module and a similarity calculation module; the model construction and training module is used to construct a neural network model, and train the network model parameters until the model parameters converge and then fix the network model parameters; the feature extraction module is used to extract features and form a protection video feature matrix to store in a database, and the similarity calculation module is used to compare whether the retrieved video is infringing.

[0039] Example 2

[0040] This embodiment relates to a homologous video retrieval method based on temporal features, which includes the following steps:

[0041] (1) First, a neural network model is constructed in the model construction and training module, and the network model parameters are trained using open source datasets such as VCDB, FIVR, and CC_WEB_VIDEO. The network model parameters are fixed after the training is completed.

[0042] (2) In the feature extraction module, the fixed parameter model is used to extract features of the video to be protected offline and form a protection video feature matrix which is stored in the database.

[0043] (3) When initiating detection on a video that needs to be retrieved to determine whether it contains suspected infringement, the same model is used to extract feature vectors from the retrieved video and the similarity is calculated with the feature matrix of protected videos in the library. The similarity calculation module compares the two videos one by one to see if they exceed the protection threshold, and returns the video pairs that exceed the threshold as the detection result.

[0044] In the model building and training module, that is, in step (1) above,

[0045] (1-1) First, extract key frames from the videos in the dataset: extract one frame every second from all videos, set the frame sampling value to 20, and sample 20 frames of equal distance from all extracted frames as the key frames of the video. Perform preprocessing on the key frame images, including random cropping, random flipping, transforming to tensors, scaling to a uniform size, etc. in the training phase. After training, the model only needs to be scaled to a uniform size when used in the feature extraction module.

[0046] (1-2) Extract features from video key frames: Construct a deep residual convolutional neural network. The specific structure of the network is as follows: Figure 2As shown in the figure, the ResNet50 network structure with pre-trained parameters on the large-scale dataset ImageNet is used to extract feature vectors from frame images. After passing through the deep network, the feature vector contains high-level semantic feature information (such as objects, scenes, etc.), and at the same time, because of the residual addition operation, the underlying semantic feature information (such as color, texture, contour, etc.) is also retained. After the model forward inference, the output of the last layer of the model is selected, and the frame image is extracted into a feature tensor with a dimension of (7, 7, 2048), where 7x7 is the length and width of the feature map in pixels, and 2048 is the number of channels.

[0047] Since infringers often try to evade existing detection methods by inserting interference frames into the original video, cutting out clips and inserting them into their own videos in a different order, the present invention Figure 3 As shown, the feature tensor extracted by the deep residual convolutional neural network is further analyzed through the two submodules of temporal feature extraction and frame importance information extraction to solve the above problem. The frame feature tensor obtained after the above steps is copied into two copies and input into the submodules of temporal feature extraction and frame importance information extraction respectively.

[0048] (1-3) Extracting timing information from key frame features: First, all key frame feature tensors of a video are arranged in the order in which they appear in the video, and are linearly transformed into one-dimensional feature vectors. A temporal coding vector is constructed, which is encoded as feature vectors of frames 1 to 20, corresponding to frames 1 to 20. The original feature vector is added to the coding vector and input into the Transformer encoder. The encoder consists of multiple layers of repeated operations, each of which includes multi-head attention mechanism, residual addition regularization, fully connected layer and other operations. After the encoder outputs, a one-dimensional feature vector is obtained, which contains the temporal feature information of the feature tensor corresponding to each frame.

[0049] (1-4) Extracting importance information from key frame features: Input all key frame feature tensors of the input video into the importance information extraction module. This submodule consists of a 3-head attention mechanism, each of which is based on a 1-dimensional convolution operation and then scaled using a softmax function. After the attention mechanism and the feature tensor are multiplied and added, a vector with a weighted sum of 1 in each dimension is obtained, which corresponds to the information importance of each frame. In the detection scenario where the infringer inserts interference frames, this method can effectively extract the information importance of each frame. After training, the model parameters can pay more attention to the frames from the original video that needs to be protected, while the weights of the interference frames inserted by the infringer will be relatively low.

[0050] (1-5) Aggregate the timing information vector and the importance information vector: After the above two submodules, two vectors are obtained, which contain the timing feature information and importance feature information of each frame respectively. The two vectors are averaged in each dimension to obtain the final weighted vector of each frame. The vector is multiplied with the video key frame feature tensor to obtain the feature vector representation of the entire video, which contains the high-level and low-level semantic information of each frame, and contains the timing relationship and importance comparison information between frames.

[0051] (1-6) Construct positive and negative sample triplets and twin network training model: When training the network model parameters, for each round of an input video, randomly sample one positive sample from other videos with the same source as the positive sample, and randomly sample one negative sample from other videos with different sources as the negative sample to form a triplet. Input the positive sample and the negative sample into two twin networks to obtain the corresponding feature vector representation, and calculate the L2 (Euclidean) distance between the feature vectors of the positive sample and the negative sample in the semantic space for the feature vector of the input video of the current training round. The distance calculation formula is:

[0052]

[0053] Among them, Q and R are the feature vectors of the two videos. In the fixed parameter model used in the feature extraction module, Q and R correspond to the video feature vectors for protection and retrieval respectively. Set a spacing threshold and construct a triplet loss function:

[0054] L(Q,R)=max{0,γ+Dist(Q,R - )-Dist(Q,R + )},

[0055] The difference between the distance between the feature vector of the current training video and the positive sample video and the distance between the corresponding negative sample is at least the threshold, which is generally set to 0.5. After training by this method, the similarity between the feature vectors of the same source videos extracted by the network model parameters is larger, and vice versa.

[0056] Use open source datasets to train network parameters until convergence to obtain a network model with fixed parameters.

[0057] In the feature extraction module, i.e., in step (2) above,

[0058] (2-1) Offline construction of a protection video feature library: For the original video works that users need to protect, they are input one by one into the fixed parameter network model obtained after the model construction and training modules are completed to obtain the corresponding protection video feature vector. The feature vectors of different protection videos are aligned according to the dimensions to construct a protection video feature matrix, and the matrix is ​​stored in the local database. This step can be completed offline, and when a new user uploads a protection video, the newly added feature vector is placed under the existing protection matrix for update.

[0059] (2-2) Extracting the retrieval video feature vector online. For the retrieval video that needs to be detected to see if it infringes the user protection video, it is similarly input into the fixed parameter network model obtained in the model building and training module to obtain the corresponding retrieval video feature vector.

[0060] In the similarity calculation module, and in the above step (3):

[0061] (3-1) After obtaining the protection video feature matrix and the search video feature vector in the feature extraction module, the search video feature vector is subjected to a leading operation, and then the two are dot-multiplied to obtain a similarity vector. Each dimension of the similarity vector represents the cosine distance of the search video feature vector to each protection video feature vector, that is, the similarity of the search video to each protection video.

[0062] (3-2) Whether the comparison exceeds the threshold: Set the similarity detection threshold, which is generally set to 0.8. For the similarity vector obtained in step (1), check whether the similarity of each dimension exceeds the above threshold one by one. If it exceeds the threshold, the similarity information and protection video information corresponding to the dimension exceeding the threshold are extracted and constructed into a video pair with the searched video, and the result of the searched suspected infringing video pair is returned. If no dimension exceeding the threshold is detected, the detection result that no suspected infringement is found is returned.

[0063] Although the specific embodiments of the present invention are described in detail above, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the purpose of the present invention, and modifications or deformations without creative labor are still within the protection scope of the present invention.

Claims

1. A homologous video retrieval method based on temporal features, characterized in that: It includes the following steps: (1) Build a neural network model and use open source data sets to train the network model parameters until the model parameters converge and then fix the network model parameters; (2) Use the trained model to extract the corresponding feature vectors for the videos that need to be protected and retrieved; (3) Setting a similarity search threshold and performing similarity calculation; The step (1) specifically comprises the following steps: (1-1) Key frame extraction: query and retrieve the video into frames, extract a certain number of frames within a certain duration according to the video length, and finally sample the extracted frames and scale them to a set size to obtain the video key frames; (1-2) Frame feature extraction: Input the key frames of the video into the deep residual convolutional neural network pre-trained on the ImageNet dataset, perform forward propagation to extract features from each channel of the frame image, and extract the vector output by the last layer of average pooling after pooling, activation, and regularization as the feature vector of each frame; (1-3) Temporal feature extraction: Input the feature vector of each frame into the Transformer encoder in the temporal order of the frames appearing in the video to obtain the temporal information feature vector between frames; (1-4) Frame importance information extraction: Use the attention mechanism to extract the mapping of different semantic spaces from the feature vectors of each frame, obtain the feature vector containing the importance information of each frame, and obtain the frame importance information feature vector after splicing and aggregation; (1-5) Feature aggregation: The output feature vectors of the self-attention mechanism and the temporal feature extraction module are input into the fully connected layer for weighted aggregation to obtain the overall feature vector representation of the video; (1-6) Twin network training: When training network parameters, in each training round, for each input video, a video with the same source as the video is extracted as a positive sample, and a video with a different source from the video is extracted as a negative sample. The three samples are constructed into a triplet, and the loss value is obtained using the triplet loss function. The gradient is calculated using the loss value, and then the neural network is fed back and propagated to update the parameters of each layer. This step is repeated until the network parameters converge.

2. The method for retrieving homologous videos based on temporal features according to claim 1, characterized in that: Step (2) specifically includes the following steps: (2-1) Constructing a video database that needs to be protected: The user-generated videos that the system needs to protect are stored in the database. The feature vector of each protected video is obtained through step (1). The feature vectors of all videos are stored in the database to obtain the feature matrix of all protected videos. (2-2) Extracting features from the retrieved video: For videos that need to be detected to determine whether they infringe the rights of the videos in the protected video library, extract the corresponding video feature vector through step (1) and perform a comparative search.

3. The method for retrieving homologous videos based on temporal features according to claim 1, characterized in that: Step (3) specifically includes the following steps: (3-1) Setting a similarity search threshold: Calculate the similarity between the searched video and the protected video. The similarity is in the range of (0,1), where 1 means that the model determines that the searched video and the protected video are 100% similar, 0 means that they are completely dissimilar, and other values ​​represent the percentage of similarity. If a search threshold is set, the system will return video pairs with similarities exceeding the threshold as suspected infringement results for the user to make further judgments later. (3-2) Calculating similarity: The feature vectors of the retrieved video obtained in step (2) are point-multiplied with the feature matrix in the protected video library one by one to obtain the cosine distance between the feature vectors of the retrieved video and each protected video as the similarity; the similarities are compared one by one to determine whether any of them exceeds a set threshold. If so, the video pair is returned as the system detection result; if not, the suspected infringing video is returned as the system detection result.

4. A homologous video retrieval system based on temporal features, the system being used in any homologous video retrieval method based on temporal features in claims 1-3, characterized in that: It includes The model building and training module is used to build a neural network model and train the network model parameters until the model parameters converge and then fix the network model parameters; A feature extraction module extracts features and forms a protection video feature matrix and stores it in a database; The similarity calculation module is used to compare and retrieve videos to see if they infringe copyright.

Images