A compact feature based local near-duplicate video detection method and system

Abstract

The application discloses a local near-duplicate video detection method and system based on compact features, which first uniformly samples video frames of a query video and database videos, and extracts deep features of the video frames as initial features; secondly, since main contents of near-duplicate frames are basically in the center of pictures, a two-dimensional Gaussian kernel is used to weight the initial features of the frames so that attention is concentrated on the center of the pictures, then some feature dimension reduction methods are used to reduce the dimension of the initial features to obtain low-dimensional compact feature vectors; then, a key frame extraction algorithm based on a video self-similarity matrix is used to extract key frames of all the videos; finally, based on the key frames of the videos extracted in the last step, a time alignment algorithm based on a graph network is used to detect and locate near-duplicate video segments.

Description

Technical Field

[0001] The present invention belongs to the field of multimedia information processing, and specifically relates to a local near-duplicate video detection method and system based on compact features, and more specifically to a video key frame selection algorithm based on a video self-similarity matrix and a video frame feature encoding method based on a Transformer. Background Art

[0002] With the rapid development of the internet, the primary way to access information has shifted from text to video. While the massive amount of online videos brings convenience, it also creates numerous problems. For example, some self-media platforms edit popular videos and republish them to gain traffic and profit, seriously infringing on the rights of the original video creators. Some pirated content producers also use online platforms to sell pirated videos and profit illegally. Due to the sheer volume and inherent complexity of these videos, these near-duplicate videos are difficult to identify manually. With the rapid development of computer vision technology in the video field in recent years, near-duplicate video detection algorithms based on video semantics have received increasing attention.

[0003] A near-duplicate video is a video that is similar to the original video after undergoing a series of transformations. These transformations include, but are not limited to, video format conversion, optical parameter (RGB, brightness) conversion, geometric parameter (mirroring, flipping) conversion, editing operations (inserting subtitles, logos, borders, etc.), slow motion or speed conversion, and so on. Generally speaking, determining whether a pair of videos are near-duplicates from a holistic perspective is called global near-duplicate video detection. In real-world network environments, global near-duplicate videos are rare. More often, certain segments within a pair of videos are near-duplicates. The algorithm that detects these segments is called a local near-duplicate detection algorithm.

[0004] Current local near-duplicate video detection algorithms are basically composed of four main modules, namely feature extraction module, feature encoding module, key frame selection module and time alignment module. Researchers have made improvements and innovations for different modules, making the local near-duplicate video detection algorithm in various forms. For example, for the feature extraction module and the feature encoding module, in one method (He S, Yang X, Jiang C, et al. A Large-scale Comprehensive Dataset and Copy-overlap Aware Evaluation Protocol for Segment-level Video Copy Detection [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2022:21086-21095.), the authors extracted multiple features of video frames and used different encoding methods to encode the features, such as ViSil features that fuse multiple convolutional intermediate layer features, R-MAC features that consider more receptive fields of pictures, and so on. The authors verified the performance differences of these features in previous local near-duplicate detection algorithms and proved the good performance of ViSil features. However, due to the large dimension of ViSil features, it has very high requirements for the storage and computing capacity of the system, so this feature is not suitable for large-scale local near-duplicate video detection.For example, for the key frame selection module and the time alignment module, in another method (Chen Jiang, Kaiming Huang, Sifeng He, Xudong Yang, Wei Zhang, Xiaobo Zhang, Yun Cheng, Lei Yang, Qing Wang, Furong Xu, et al. Learning segment similarity and alignment in large-scale content based video retrieval[C] / / In Proceedings of the 29th ACM International Conference on Multimedia, pages 1618–1626, 2021.1, 2, 3, 4, 7.), the authors proposed a local near-repeated video detection network that can be trained end-to-end. The network integrates an efficient self-supervised key frame extraction network SKE and a highly robust time alignment network SPD based on near-repeated pattern detection. The SKE module can effectively reduce redundant frames, improve the quality of video key frames, and make the video semantic information clearer, while the SPD module can effectively detect and identify multiple near-repeated patterns, making the system more robust. Although this method has achieved the best results among the current local near-duplicate video detection algorithms, it requires the use of ultra-large-scale datasets for training and has a large feature dimension, so it also has some limitations. Summary of the Invention

[0005] The purpose of the present invention is to overcome the shortcomings of the above-mentioned prior art and provide a local near-duplicate video detection method and system based on compact features to solve the problems in the prior art of local near-duplicate video detection algorithms that have high requirements on the system's storage and computing capabilities, or have complex processes and limited scope of use.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] A local near-duplicate video detection method based on compact features, comprising:

[0008] Sample the query video and the database video to be retrieved to obtain the sampled frames of each video; extract the deep features of the sampled frames as the initial features;

[0009] Add spatial attention based on a two-dimensional Gaussian kernel to each initial feature and perform R-MAC processing to obtain R-MAC features of the weighted initial features; the R-MAC features are passed through a Transformer feature encoder to generate low-dimensional compact features of the initial features; the Transformer feature encoder is obtained by training a twin network, and the Transformer feature encoder includes a backbone network with shared training weights and a mapping multi-layer perceptron;

[0010] Based on low-dimensional compact features, the self-similarity matrix of each video is obtained. After visualization and thresholding of the self-similarity matrix, the key frames of each video are obtained through object detection network detection.

[0011] Based on the key frames of the query video and the key frames of the database video to be detected, the duplicate videos are detected through the time alignment algorithm based on the graph network, and the location of the duplicate videos is obtained at the same time.

[0012] A further improvement of the present invention is:

[0013] Preferably, initial features are extracted from the sampled frames by a deep convolutional neural network.

[0014] Preferably, the process of adding spatial attention based on a two-dimensional Gaussian kernel to each initial feature and performing R-MAC processing is as follows: the features of each channel on the initial features output by the neural network are weighted using a two-dimensional Gaussian kernel to obtain weighted initial features, and then the weighted initial features are activated to the maximum extent by regional convolution to obtain R-MAC features of the weighted initial features.

[0015] Preferably, the low-dimensional compact feature is:

[0016]

[0017] Among them, L 2 (*) represents l2 normalization; F encoder is the regional feature sequence after Transformer feature encoding, and the calculation formula is:

[0018]

[0019] Among them, Q is the query matrix, K is the key matrix, V is the value matrix, S 2 is the number of regional eigenvectors.

[0020] Preferably, the process of training the feature encoder through the twin network is: training the backbone network and the mapping multilayer perceptron of the Transformer feature encoder through the twin network;

[0021] In the training of the backbone network and the mapping multi-layer perceptron, the initial features of the anchor sample and the initial features of the positive sample are input into two branches of the twin network, and the symmetric negative cosine similarity of the two branches is minimized, which is the loss function of the network, and the training is completed when the loss function converges.

[0022] Preferably, the initial features of the anchor sample are taken from the anchor sample, and the initial features of the positive sample are taken from the positive sample; the anchor sample and the positive sample are taken from the repeated frame dataset; the repeated frame dataset generation process is as follows: according to the VCDB dataset label, all near-duplicate segment pairs are obtained, the anchor sample set and the positive sample set are obtained after sampling the near-duplicate segment pairs in the dataset, and after removing part of the repeated images from each sample set, the low-dimensional features of all frames in the anchor sample set and the positive sample set are extracted to construct a similarity matrix; a similarity threshold is set, and the similarity matrix is labeled by the similarity threshold to obtain the repeated frame dataset.

[0023] Preferably, the key frame acquisition process of the video is as follows: the self-similarity matrix is:

[0024] ma=f(f T )∈R N×N (8)

[0025] wherein, f∈R N×C is the low-dimensional compact feature sequence of the video frame, N is the frame number, and C is the feature dimension.

[0026] The thresholding process is as follows: the points with a similarity less than the threshold value 0.9 in the self-similarity matrix are assigned to 0, otherwise no processing is performed;

[0027] After the size of the thresholding processed self-similarity matrix is adjusted to 640x640 pixels, it is input into the target detection network, the target detection network detects the similar frame pattern in the self-similarity matrix, and the key frame of the video is selected according to the similar frame pattern.

[0028] Preferably, based on the key frame of the query video, candidate key frames are selected from the key frames of the database video to be detected, and the similarity between the candidate key frames and the key frame of the query video is greater than a threshold value.

[0029] Then, the key frame of the query video and the candidate key frame are input into the graph network based time alignment algorithm to detect the repeated video and obtain the position of the repeated video.

[0030] Preferably, the graph network based time alignment algorithm satisfies the following constraint conditions:

[0031] (a) 0 < j-i < wnd;

[0032] (b) 0 < TS(R j,n )-TS(R i,m) <wnd;

[0033]

[0034] (d)w(e mn )>θ t ;

[0035] (e)L≥L min ; (17)

[0036] Among them, i and j are key frame nodes, and i <j;R i,m represents the mth candidate frame of the i-th key frame of the query video, R j,n represents the nth candidate frame of the jth key frame of the query video, R i,m and R j,n Belong to the same video, e mn Indicates that from R i,m to R j,n A directed edge of w(e mn ) represents the weight of the directed edge.

[0037] A local near-duplicate video detection system based on compact features, comprising:

[0038] The feature extraction module is used to sample the query video and the database video to be retrieved, obtain the sampled frames of each video, and extract the deep features of the sampled frames as the initial features;

[0039] A feature encoding module is used to add spatial attention based on a two-dimensional Gaussian kernel to each initial feature and perform R-MAC processing to obtain R-MAC features of the weighted initial features; the R-MAC features of the weighted initial features are passed through a Transformer encoder to generate low-dimensional compact features of the initial features; the Transformer encoder is obtained through the training of a twin network, and the Transformer encoder includes a backbone network with shared training weights and a mapping multi-layer perceptron;

[0040] The key frame extraction module is used to obtain the self-similarity matrix of each video based on low-dimensional compact features. After visualizing and thresholding the self-similarity matrix, it is detected through the object detection network to obtain the key frame of each video.

[0041] The near-duplicate segment detection module is used to detect duplicate videos based on the key frames of the query video and the key frames of the database video to be detected through a graph network-based time alignment algorithm, and at the same time obtain the location of the duplicate videos.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] The present invention discloses a local near-duplicate video detection method based on compact features. The method first uniformly samples the query video and the database video and extracts deep features as initial features. Secondly, since the main content of the near-duplicate frame is basically in the center of the picture, the initial features of the frame are weighted using a two-dimensional Gaussian kernel to focus on the center of the picture. Then, some feature dimensionality reduction methods are used to reduce the initial feature dimension to obtain a low-dimensional compact feature vector. Then, a key frame extraction algorithm based on the video self-similarity matrix is ​​used to extract the key frames of all videos. Finally, based on the video key frames extracted in the previous step, a time alignment algorithm based on a graph network is used to detect and locate near-duplicate video clips. This method is based on a Transformer feature encoder, which learns the subtle differences between a large number of near-duplicate frame features. Therefore, it can cluster near-duplicate frame features in the feature space and exclude non-near-duplicate frame features, improving the expressiveness of the encoded features and enhancing robustness. It can also effectively reduce the feature dimension and improve the efficiency of the algorithm. A key frame extraction algorithm based on the video self-similarity matrix can obtain rich but non-redundant key frames, so that the key frame feature sequence can more comprehensively describe the original video content, improving the algorithm performance. In addition, it also significantly reduces the overhead of storing and calculating redundant key frame features. The Transformer feature encoder is trained through a twin network. The network does not require a large number of negative samples to effectively learn the similar semantic information between near-duplicate frames. Therefore, there is no need for the heavy and difficult work of hard negative sample labeling, making the training process simpler and more efficient. This method effectively solves the problems of existing local near-duplicate video detection algorithms, such as high feature storage consumption, low overall query efficiency, and inability to match near-duplicate segments based on frame-level semantic information.

[0044] Furthermore, the present invention adopts the deep features of Transformer encoding frames. Compared with traditional features and deep features, this feature has better representation and robustness. At the same time, the feature dimension is low, and the computational and storage overheads consumed are greatly reduced, thereby effectively improving the performance and efficiency of the algorithm.

[0045] Furthermore, the present invention uses a new key frame extraction algorithm based on the video self-similarity matrix, which can extract rich but non-redundant key frames, so that the key frames can better describe the original video content, thereby effectively improving the detection accuracy while reducing storage and computing overhead.

[0046] The present invention also discloses a local near-duplicate video detection system based on compact features, which includes a feature extraction module, a feature encoding module, a key frame extraction module and a near-duplicate segment detection module, and the above method is implemented through the above units. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is the overall flow chart of the present invention.

[0048] Figure 2 This is a training flow chart of the feature encoder in the present invention.

[0049] Figure 3 Generate a flow chart for the near-duplicate frame dataset.

[0050] Figure 4 Schematic diagram of the key frame extraction module based on video self-similarity matrix.

[0051] Figure 5 This is an example diagram of similar frame patterns in the video self-similarity matrix.

[0052] Figure 6 This is a schematic diagram of the time alignment algorithm based on graph network.

[0053] Figure 7 This is a PR curve diagram of the algorithm of the present invention using different encoding methods and performing 5-fold cross validation on the VCDB core set. DETAILED DESCRIPTION

[0054] The present invention is described in further detail below with reference to the accompanying drawings:

[0055] In the description of the present invention, it should be noted that the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the devices or components referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention; the terms "first", "second", and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance; in addition, unless otherwise expressly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection or a detachable connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be a communication between the internal parts of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0056] The present invention discloses a local near-duplicate video detection method based on compact features. The method first uniformly samples video frames of the query video and the database video, and extracts their deep features as initial features; secondly, since the main content of the near-duplicate frame is basically in the center of the picture, a two-dimensional Gaussian kernel is used to weight the initial features of the frame to focus on the center of the picture, and then some feature dimensionality reduction methods are used to reduce the initial feature dimension to obtain a low-dimensional compact feature vector; then a key frame extraction algorithm based on the video self-similarity matrix is ​​used to extract the key frames of all videos; finally, based on the video key frames extracted in the previous step, a time alignment algorithm based on a graph network is used to detect and locate the near-duplicate video clips.

[0057] The following steps are involved:

[0058] Step 1: Uniformly sample video frames:

[0059] Each query video and the database video to be retrieved are sampled at a frame rate of 2 frames per second to obtain a sampling frame of each video, where the video includes the query video and the database video to be retrieved.

[0060] Step 2: Extract initial features of the frame:

[0061] Feed the sampled frames in step 1 into the deep convolutional neural network to extract their deep features as initial features;

[0062] Step 3: Encode initial features:

[0063] Add a spatial attention based on a two-dimensional Gaussian kernel to the initial features extracted in step 2, and perform R-MAC processing. Then use the feature encoder to encode the R-MAC features of the weighted initial features into low-dimensional compact features.

[0064] The specific process of encoding the initial features of the frame is as follows:

[0065] Step 3.1: First, add a spatial attention based on a two-dimensional Gaussian kernel to the initial features in step 2. The two-dimensional Gaussian kernel weight calculation formula is:

[0066]

[0067] Where w(x,y) represents the weight of the pixel point (x,y) on the initial feature, W and H represent the width and height of the initial feature respectively, σ x and σ y is the parameter of the two-dimensional Gaussian kernel. The initial features of the neural network output The features on each channel After weighting with two-dimensional Gaussian kernel, we can obtain weighted initial features, which are weighted channel features:

[0068]

[0069] Where l = 1, 2, ... C, then the corresponding initial features of the frame with the added spatial attention are:

[0070]

[0071] Step 3.2: Then, use the Region-Maximum Activation of Convolutions (R-MAC) method to encode the weighted initial features in step 3.1, that is, F l w Divide into multiple equal and partially overlapping areas Where w=h=2min(W,H) / (S+1), i,j=1,2,...,S, and a maximum pooling operation is performed on each channel of each regional feature, and a regional feature vector of dimension C can be obtained. For the entire weighted initial feature, the number of regional feature vectors is S×S, and the R-MAC feature of the weighted initial feature is obtained:

[0072]

[0073] Where S = 1, 2, 3..., the regional feature vector

[0074] Step 3.3: Flatten the R-MAC features obtained in step 3.2 into a sequence of regional feature vectors Then it is fed into the trained Transformer feature encoder to generate low-dimensional compact features of the initial features. Assume that the parameter matrices of the encoder are W Q ,W K ,W V , then the query matrix Q, key matrix K, and value matrix V of the input sequence can be obtained through three linear transformations, namely:

[0075]

[0076] The encoded regional feature sequence is:

[0077]

[0078] F encoder After superposition in each dimension and then l2 normalization, the low-dimensional compact features of the initial features can be obtained, which are called T-RMAC features in this invention:

[0079]

[0080] Among them, L 2(*) represents l2 normalization, represents a natural number vector of dimension C, The representative dimension is S 2 ×C natural number matrix.

[0081] Step 4: Extract keyframes:

[0082] A key frame extraction algorithm based on the video self-similarity matrix is ​​used to extract the key frames of each video and save their low-dimensional compact features. The specific steps for extracting key frames based on the video self-similarity matrix are as follows:

[0083] Step 4.1: Assume that a video has N frames after sampling in step 1. After steps 2 and 3, low-dimensional compact features of all frames are obtained. Then, a low-dimensional compact frame feature sequence f∈R for each video can be obtained. N×C , let ma represent the self-similarity matrix of the video frame sequence, then the self-similarity matrix of each video can be obtained:

[0084] ma=f(f T )∈R N×N (8)

[0085] Step 4.2: After visualizing and thresholding the self-similarity matrix in step 4.1, use the trained object detection network to detect similar frame patterns. For the detected similar frame patterns, take one frame as the key frame, and for the detected non-similar frame patterns, take all frames as key frames.

[0086] Step 5: Detection and localization of near-duplicate video segments:

[0087] Based on the key frame features of the query video and the database video to be retrieved obtained in step 4, a graph network-based time alignment algorithm is used to detect and locate near-duplicate segments. The specific steps are as follows:

[0088] Step 5.1: In step 4, the low-dimensional compact features of all video key frames are obtained. For each key frame of the query video, search in the database and select the top K key frames whose cosine similarity with the key frame is greater than the threshold as candidate key frames.

[0089] Step 5.2: Based on the low-dimensional compact features of the query video's keyframes and their candidate keyframes, a graph network-based time alignment algorithm is used to locate near-duplicate segments. Each candidate keyframe is used as a node in the graph network, and candidate keyframes belonging to the same video segment establish directed edges based on their timestamps. The weight of the directed edge is the similarity between the candidate keyframe and the query keyframe. Then, locating near-duplicate segments can be converted into finding the path with the largest sum of directed edge weights in the graph network. This path is the best near-duplicate matching segment.

[0090] The following is a further description with reference to specific drawings and embodiments.

[0091] Regarding the local near-duplicate video detection method based on compact features provided by the present invention, taking a query video as an example, the specific process of passing through each module and the specific implementation method of each module are further described:

[0092] Step 1: Uniformly sample video frames:

[0093] In this method, the query video and the database video are densely and evenly sampled at a frame rate of two frames per second. The purpose is to avoid the performance degradation of the algorithm caused by missing key frames. At this time, for a certain video, its sampling frame set Fr={Fr1,Fr2,...,Fr N}, N is the number of sampling frames.

[0094] Step 2: Extract initial features of the sampling frame:

[0095] Normalize the N sample frames of the video in step 1 and resize each frame to 224×224 pixels. Then, feed the normalized N sample frames into the VGG-16 network pre-trained on the ImageNet dataset and extract the output of the last convolutional layer of VGG-16 as the initial feature of the frame. The initial feature dimension is 14×14×512. At this time, the initial feature set F = {F1, F2, ..., F N}.

[0096] Step 3: Encode initial features:

[0097] First, add spatial attention based on a two-dimensional Gaussian kernel to each initial feature in the initial feature set in step 2. The parameter σ of the two-dimensional Gaussian kernel is x and σ y According to the statistical "3σ rule", it is set to one-third of the distance from the center to the edge. At this time, the weighted initial feature set can be obtained The weighted formula refers to the detailed description of this part above; then all the weighted initial features in the set are encoded using the regional convolutional maximum activation coding (R-MAC) method. Here, the regional scale parameter S in the R-MAC method is set to 3. At this time, the R-MAC feature set of the weighted initial features can be obtained, corresponding to each element in the set Then all R-MAC features in the set are flattened into a regional feature vector sequence and input into the trained Transformer encoder network to generate T-RMAC features. At this time, the low-dimensional feature sequence of the video is obtained. It should be noted that the output of the encoder needs to be normalized by l2 before it becomes the T-RMAC feature.

[0098] The training process of the Transformer feature encoder is as follows Figure 2 As shown in the figure, the present invention designs a twin network (i.e., a neural network composed of two neural networks with identical structures and shared weights) to train the Transformer feature encoder e. Specifically, during the training process, the initial features of the anchor sample and the initial features of the positive sample in a pair of near-duplicate frames are first used as input to the two branches of the twin network, where the anchor sample and the positive sample come from a self-made near-duplicate frame dataset. Figure 2 The Transformer feature encoder e consists of a backbone network (Transformerencoder) and a projection multilayer perceptron (projectionMLPhead). When the backbone network and the projection multilayer perceptron are trained through the twin network, the encoders of the two branches in the twin network share the training weights. A prediction multilayer perceptron denoted as h will convert the encoding results of the anchor sample features and match the encoding results of the positive sample features. Assuming that p1 and z2 are the outputs of the two branches, the role of the network is to minimize their negative cosine similarity D(p1,z2):

[0099]

[0100] In the above formula, ·2 represents the L2 norm of the vector, and a symmetric loss function can be defined as:

[0101]

[0102] Since the stop-gradient operation is the key point for the network to work, the above two formulas can be adjusted to the following form:

[0103] D(p1,z2)=D(p1,stopgrad(z2))(11)

[0104]

[0105] Therefore, the purpose of the network is to minimize the loss function Loss, where Loss∈[-1,0].

[0106] The generation process of the near-duplicate frame dataset mentioned above is as follows Figure 3 As shown, based on the annotations of near-duplicate segments in the VCDB dataset (near-duplicate video detection dataset), all near-duplicate segments in the database to be retrieved are obtained. To avoid missing key frames, dense sampling is adopted for all near-duplicate segments. Therefore, a sampling baseline value is set, which is 2 frames per second in this invention. Since most near-duplicate segments have the same duration, they are all sampled according to the sampling baseline value. For near-duplicate segments with double-speed or slow speed, the sampling baseline value is adjusted according to the double-speed rate. In this way, for each pair of near-duplicate segments, an anchor sample set and a positive sample set of sampled frames are obtained. Then, a simple deduplication algorithm (such as a hash-aware image deduplication algorithm) is used in each set to remove most of the simple duplicate images; thereafter, low-dimensional features of all frames in the anchor sample set and the positive sample set are extracted (the present invention uses the output features of the last convolutional layer of VGG-16 as the deep features of the image and further uses the maximum activation convolution method to reduce the dimension into a feature vector), and a similarity matrix is ​​constructed, where the number of rows represents the number of frames in the anchor sample set after deduplication (assuming it is m), the number of columns represents the number of frames in the positive sample set after deduplication (assuming it is n), and the value of a pixel point (i, j) in the matrix represents the feature of the i-th frame in the anchor sample set and the similarity matrix. The similarity of the features of the j-th frame in the positive sample set; finally, for a certain row x in the matrix, a maximum value can be found. Assuming it is in the y-th column, the coordinates of this maximum value represent that the x-th frame in the anchor sample set and the y-th frame in the positive sample set may be near-duplicate frames. Recording (x, y) is a labeling. Since there may be a situation where there are no near-duplicate frames in a certain row, a similarity threshold is set. Labeling is performed only when the similarity is greater than this threshold. Otherwise, no labeling is performed. In this embodiment, after the labeling is completed, the entire data set has a total of about 320,000 pairs of near-duplicate frames, including 160,000 corresponding anchor samples and 160,000 positive samples.

[0107] Step 4: Extract keyframes:

[0108] like Figure 6 The following is a schematic diagram of the key frame extraction module based on the video self-similarity matrix. In step 3, a low-dimensional compact feature sequence of the video is obtained. Transposing the sequence and multiplying it with its own matrix can obtain the self-similarity matrix of the video For a certain point (x, y) in the matrix, its value represents the value of the inner product of the x-th low-dimensional feature and the y-th low-dimensional feature in the low-dimensional feature sequence. Since the low-dimensional features in the sequence are l2-normalized, this value is equivalent to their cosine similarity:

[0109]

[0110] Where 0≤x,y≤N.

[0111] Then the self-similarity matrix of the video is calculated as Figure 5 The color bar visualization shown, in order to more accurately identify similar frame patterns and dissimilar frame patterns on the diagonal line, the self-similarity matrix is ​​thresholded, that is, points with a similarity less than 0.9 are assigned a value of 0, otherwise no processing is performed, and the processed similarity matrix is ​​resized to 640×640 pixels and sent to the trained target detection network for similar frame pattern detection. Finally, for the detected similar frame pattern, one frame is taken as the key frame, and for the dissimilar frame pattern, all frames are taken as key frames, and finally a low-dimensional compact frame feature sequence of the video is obtained. Where M represents the number of key frames in the video.

[0112] Specifically, the object detection network training process first randomly selected 1,000 videos from the VCDB background set, and obtained their self-similarity matrices in a manner consistent with the algorithm of the present invention, and distributed them into training, validation, and test sets in a ratio of 8:1:1. Then, the dataset was annotated using an algorithm and manual method. That is, all possible similar frame patterns were first annotated using a scene detection algorithm, and then the annotations were manually fine-tuned and selected. Finally, the self-similarity matrix was set to a 640×640 image and fed into the lightweight object detection model ppyolov2 for training. The model was trained for 250 epochs with a batch size of 4.

[0113] Step 5: Detection and localization of near-duplicate video segments:

[0114] like Figure 1 As shown in FIG, the detection module of near-duplicate video segments consists of two steps, namely, candidate key frame selection and time alignment. The specific processes of the two steps are described in detail below.

[0115] Step 1: Candidate frame selection:

[0116] In step 4, a low-dimensional compact frame feature sequence of the video is obtained. The video can be represented by this feature sequence. In order to achieve fast retrieval and matching of feature vectors, the present invention uses cosine similarity to measure the similarity between two key frame features. The calculation formula is as follows:

[0117]

[0118] Where f1 and f2 represent the feature vectors of two key frames, f1 i and f2 i Denote the i-th dimension of feature vectors f1 and f2 respectively, then sim(f1,f2)∈[0,1]. For each key frame of the query video, search in the database and select the key frames whose similarity with the key frame is greater than a certain threshold θ. t The first K key frames of the query video are used as candidate key frames. In the present invention, this value is set to 30. Therefore, for each key frame of the query video, a candidate frame sequence with a number less than or equal to K is retrieved. Since the search is performed in the entire video database, the frames in the candidate frame sequence do not necessarily come from the same video.

[0119] Step 2: Time alignment based on graph network:

[0120] After obtaining the key frame features of the video and their candidate frame features, the present invention uses time alignment based on the graph network to realize the location of near-duplicate segments. Each candidate frame retrieved by cosine similarity matching is used as a node in the graph network. The candidate frames belonging to the same video segment establish directed edges according to their timestamps, such as Figure 6 The figure shows the search results for a query video frame sequence consisting of five keyframes and its candidate frames. Two virtual nodes, the start node and the end node, are added when constructing the graph network. This ensures that all paths in the graph begin at the start node and end at the end node, ensuring that every possible path in the graph is mined. The weight of directed edges depends on the similarity between the candidate frame and the query frame. When the destination node is the end node, the edge weight is 0. The following mathematical formulas describe the specific details of the algorithm.

[0121] Let Q = [q1,q2,...,q X ] represents the query video key frame sequence; D = [d1, d2, ..., d Y ] represents the database key frame sequence; A Q =[a1,a2,...,a L ] and A D =[b1,b2,...,b L ] represent the best matching sequences of query video key frames and database video key frames respectively, then the positioning of near-duplicate segments can be expressed by the following formula:

[0122]

[0123] It is necessary to ensure that:

[0124] a i i+1 ,b i i+1 ​​,1≤a L ≤X,1≤b L ≤Y(16)

[0125] This formula shows that the best matching sequence for the query video is the video frame sequence represented by the path with the largest sum of weights among all the paths constructed in the graph network, and all paths from the start node to the end node are possible near-duplicate video frame sequences detected.

[0126] In order to improve the computational efficiency of the graph network and the accuracy of positioning, in addition to the time sequence restriction, additional constraints need to be introduced when constructing the graph network. i ,q j Respectively represent the i-th and j-th key frame nodes of the query video, and i <j;R i,m represents the mth candidate frame of the i-th key frame of the query video, R j,n represents the nth candidate frame of the jth query frame of the query video, R x,k represents the kth candidate frame of the xth query frame of the query video, R i,m and R j,n Belong to the same video, TS(no) represents the timestamp of node no; e mn Indicates that from R i,m to R j,n A directed edge of w(e mn ) represents the weight of the directed edge. When constructing a graph network, the following constraints need to be met:

[0127]

[0128] Among them, constraints (a) and (b) are restrictions on the time flow of nodes, that is, the two nodes connected by the directed edge must not only meet the time order of the video, but also the interval between the timestamps of the two nodes does not exceed wnd. In this invention, this value is set to 3; constraint (c) is the pruning of directed edges. For directed edges e mn , if there is a node R x,k (i <x<j),可以建立一条从R i,m to R x,k Then to R j,n The path, R x,k is the kth candidate frame of the xth query frame of the query video, then e mn It is redundant, that is, directed edges should be established between adjacent nodes that satisfy the time order as much as possible; constraint (d) is a restriction on the similarity of candidate frames. Only when the similarity of candidate frames is greater than the threshold θ t When , this frame will be considered to be added to the graph network. In this invention, this value is set to 0.8; constraint (e) is the limit on the path length L, that is, the minimum path length is Lmin The constraint condition can effectively reduce the number of edges in the graph network, greatly improving the efficiency of the algorithm, thereby ensuring the efficiency and accuracy of the near-duplicate segment location step. In the present invention, this value is set to 4.

[0129] As shown in Table 1, the results of a 5-fold cross-validation on the public dataset VCDB core set using the algorithm of the present invention and the existing local near-duplicate video detection are shown. The VCDB dataset consists of two parts: a core dataset and a background dataset. The core dataset is 528 videos collected from major video websites based on 28 search keywords, with a total duration of about 27 hours. A 5-fold cross-validation was performed on the VCDB core dataset, that is, for each fold, 23 of the 28 query sets in the VCDB core set were selected as training sets for training the feature encoder, and the remaining 5 were used as test sets. Finally, the average of the 5-fold test set results was taken as the overall result of the algorithm. The present invention uses Segment Precision (SP) and Segment Recall (SR) and their harmonic mean F1-Score (F1) as evaluation indicators. The calculation formulas for SP, SR and F1-Score are as follows:

[0130]

[0131] In the table CTE(Matthijs Douze, Revaud, Jakob J. Verbeek, Hervé Jégou, and Cordelia Schmid. Circulant tempo-ral encoding for video retrieval and temporal alignment. International Journal of Computer Vision, 119:291–306, 2015.), TMK (Sébastien Poullot, Shunsuke Tsukatani, Anh Phuong Nguyen, Hervé Jégou, and Shin'ichi Satoh.Tempo-ral Matching Kernel with Explicit Feature Maps.In ACMMultimedia 2018,pages 1–10,Brisbane,Australia,October 2015.ACM Press.),LAMV(L.Baraldi,M.Douze,R.Cucchiara,and H.Jegou.Lamv:Learning to align and match videos with kernelized temporal layers[C] / / In 2018 IEEE / CVF Conference on Computer Vision and Pattern Recognition, pages 7804–7813, 2018.), SPD (Wang KH, Cheng CC, Chen YL, et al. Attention-based deep metric learning for near-duplicate video retrieval [C] / / 2020 25th International Conference on Pattern Recognition (ICPR). IEEE, 2021: 5360-5367.) are four existing local near-duplicate video detection algorithms. It can be found that the algorithm of the present invention is slightly inferior to the SPD method using large dimensional features (ViSiL), but the feature size of the latter is 67.5 times that of the feature size of the method of the present invention, indicating that the latter algorithm requires huge computational and storage costs. Obviously, this method is unrealistic in large-scale local near-duplicate retrieval tasks; compared with other algorithms, the average F1-Score of the algorithm of the present invention reached 83.3%, which is higher than other algorithms.In addition, the sampling frame rate in Table 1 represents the number of frames sampled per second in the video. The lowest sampling frame rate of the algorithm of the present invention is only 2, which further indicates that the algorithm of this paper consumes only less computing and storage costs. Therefore, the algorithm of the present invention is more suitable for local near-duplicate detection tasks in large-scale scenarios.

[0132] Table 1 Comparison of 5-fold cross-validation experimental results of different algorithms on the VCDB core set Unit %

[0133]

[0134] like Figure 7 As shown in the figure, when using different feature encoding methods, the PR curve of the algorithm of the present invention is obtained by performing 5-fold cross validation on the VCDB core dataset. For each fold, the threshold of the minimum number of matching frames in the temporal network (L min ) to obtain the corresponding precision and recall, and finally fold 50% on each L min The precision and recall rates under the given conditions are averaged and the PR curve is plotted based on the average value. It can be seen that the algorithm of the present invention achieves a better precision rate than the other five feature encoding methods at each level of recall rate.

[0135] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A local near-duplicate video detection method based on compact features, characterized in that: include: Sample the query video and the database video to be retrieved, and obtain the sampled frames of each video; Extract the deep features of the sampled frames as initial features; Add spatial attention based on a two-dimensional Gaussian kernel to each initial feature and perform R-MAC processing to obtain R-MAC features of the weighted initial features; the R-MAC features are passed through a Transformer feature encoder to generate low-dimensional compact features of the initial features; the Transformer feature encoder is obtained by training a twin network, and the Transformer feature encoder includes a backbone network with shared training weights and a mapping multi-layer perceptron; Based on low-dimensional compact features, the self-similarity matrix of each video is obtained. After visualization and thresholding of the self-similarity matrix, the key frames of each video are obtained through object detection network detection. Based on the key frames of the query video and the key frames of the database video to be detected, the duplicate videos are detected through the time alignment algorithm based on the graph network, and the location of the duplicate videos is obtained at the same time.

2. The method for detecting local near-duplicate videos based on compact features according to claim 1, wherein: Initial features are extracted from the sampled frames through a deep convolutional neural network.

3. The method for detecting local near-duplicate videos based on compact features according to claim 1, wherein: The process of adding spatial attention based on a two-dimensional Gaussian kernel to each initial feature and performing R-MAC processing is as follows: the features of each channel on the initial features output by the neural network are weighted using a two-dimensional Gaussian kernel to obtain weighted initial features, and then the weighted initial features are activated to the maximum extent through regional convolution to obtain the R-MAC features of the weighted initial features.

4. The method for detecting local near-duplicate videos based on compact features according to claim 1, wherein: The low-dimensional compact features are: Among them, L 2 (*) represents l2 normalization; F encoder is the regional feature sequence after Transformer feature encoding, and the calculation formula is: Among them, Q is the query matrix, K is the key matrix, V is the value matrix, S 2 is the number of regional eigenvectors.

5. The method for detecting local near-duplicate videos based on compact features according to claim 1, wherein: The process of training the feature encoder through the twin network is as follows: training the backbone network of the Transformer feature encoder and the mapping multilayer perceptron through the twin network; During the training of the backbone network and the mapping multi-layer perceptron, the initial features of the anchor samples and the initial features of the positive samples are input into the two branches of the twin network to minimize the symmetric negative cosine similarity of the two branches. The symmetric negative cosine similarity is the loss function of the network. When the loss function converges, the training ends.

6. The method for detecting local near-duplicate videos based on compact features according to claim 5, characterized in that: The initial features of the anchor samples are taken from the anchor samples, and the initial features of the positive samples are taken from the positive samples. The anchor samples and positive samples are taken from the repeated frame dataset. The repeated frame dataset is generated by obtaining all near-duplicate segments according to the VCDB dataset annotation, sampling the near-duplicate segments in the dataset to obtain the anchor sample set and the positive sample set. After removing some duplicate images from each sample set, the low-dimensional features of all frames in the anchor sample set and the positive sample set are extracted to construct a similarity matrix. A similarity threshold is set, and the similarity matrix is ​​labeled using the similarity threshold to obtain a repeated frame dataset.

7. The method for detecting local near-duplicate videos based on compact features according to claim 1, wherein: The key frame acquisition process of the video is as follows: the self-similarity matrix is: at=f(f T )∈R N×N (8) Where f∈R N×C is a low-dimensional compact feature sequence of video frames, N is the number of frames, and C is the feature dimension; The thresholding process is as follows: assigning 0 to points in the self-similarity matrix whose similarity is less than a threshold of 0.9, and not processing otherwise; The thresholded self-similarity matrix is ​​resized to 640×640 pixels and then input into the object detection network. The object detection network detects similar frame patterns in the self-similarity matrix and selects the key frames of the video based on the similar frame patterns.

8. The method for detecting local near-duplicate videos based on compact features according to claim 1, wherein: Based on the key frames of the query video, candidate key frames are first selected from the key frames of the database video to be detected, and the similarity between the candidate key frames and the key frames of the query video is greater than a threshold; The key frames and candidate key frames of the query video are then input into a time alignment algorithm based on a graph network to detect duplicate videos and obtain the locations of the duplicate videos.

9. The method for detecting local near-duplicate videos based on compact features according to claim 8, wherein: The graph network-based time alignment algorithm satisfies the following constraints: (a)0 <j-i<wnd; (b)0<TS(R j,n )-TS(R i,m )<wnd; (c) (d)w(e mn )>θ t ; (e)L≥L min ; (17) Among them, i and j are key frame nodes respectively, and i <j;R i,m represents the mth candidate frame of the i-th key frame of the query video, R j,n represents the nth candidate frame of the jth key frame of the query video, R i,m and R j,n Belong to the same video, e mn Indicates that from R i,m to R j,n A directed edge of w(e mn ) represents the weight of the directed edge.

10. A local near-duplicate video detection system based on compact features, characterized in that: include: The feature extraction module is used to sample the query video and the database video to be retrieved and obtain the sample frames of each video; Extract the deep features of the sampled frames as initial features; A feature encoding module is used to add spatial attention based on a two-dimensional Gaussian kernel to each initial feature and perform R-MAC processing to obtain R-MAC features of the weighted initial features; the R-MAC features of the weighted initial features are passed through a Transformer encoder to generate low-dimensional compact features of the initial features; the Transformer encoder is obtained through the training of a twin network, and the Transformer encoder includes a backbone network with shared training weights and a mapping multi-layer perceptron; The key frame extraction module is used to obtain the self-similarity matrix of each video based on low-dimensional compact features. After visualizing and thresholding the self-similarity matrix, it is detected through the object detection network to obtain the key frame of each video. The near-duplicate segment detection module is used to detect duplicate videos based on the key frames of the query video and the key frames of the database video to be detected through a graph network-based time alignment algorithm, and at the same time obtain the location of the duplicate videos.

Images