A short video deduplication method based on deep learning

Abstract

The present invention discloses a short video deduplication method based on deep learning, including: preprocessing a target video and an inquiry video to be compared, and obtaining tensor representations of the target video and the inquiry video respectively; using a pre-trained neural network to obtain the spatiotemporal feature vectors of the target video and the inquiry video respectively; constructing a neural network model for video feature similarity discrimination and training the neural network model; inputting the spatiotemporal feature vectors of the target video and the inquiry video into the trained neural network model to obtain the similarity value of the target video and the inquiry video; using the similarity value to determine whether to delete the target video and the inquiry video. The present invention uses a video understanding algorithm to calculate video repetition for the situation of short video deduplication, storage and management, solves the problem of short video detection for repeated video content, and has fast convergence and high efficiency.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and specifically relates to a short video deduplication method based on deep learning, which can construct a video feature library under the condition of the number of videos at the short video platform level, so as to quickly and efficiently detect duplicate videos. Background Art

[0002] With the change of user habits and the improvement of the network environment, short videos are becoming more and more popular. For the audience, a good viewing experience has a lot to do with the video content. At present, the main high-quality videos across the entire network mainly come from MCN agencies. In order to cover the entire network content faster and better, some companies will choose to cooperate with content agents, and the agents will have a lot of duplicate copyrighted content, resulting in duplicate content. In addition, moving videos will also lead to duplicate content, which will cause a very poor user experience. Furthermore, a large number of videos with similar content is also a huge challenge for short video platforms to store. In order to avoid unnecessary waste of resources, it is very necessary to deduplicate video content.

[0003] At present, there are two main methods for video similarity comparison: (1) Generate the key frames of the video through a certain algorithm, then use some image feature extraction algorithms to calculate the features of the picture, fuse these features to generate video features, and finally generate video similarity by matching the video features. Feature extraction methods are divided into traditional manual feature extraction methods and deep learning methods. Liu Shouqun et al. used the SIFT algorithm and LSH (locality-sensetive hashing) to generate matches between video frames to quickly detect repeated clips between videos. Feature matching is divided into machine learning and deep learning methods. Song Xiaokang et al. selected features from the repeated video feature database and combined them into triplets to train the twin neural network to further reduce the distance between repeated videos and reduce the dimension of the features. (2) Abandoning the key frame algorithm, the video understanding model (such as I3D, Inflated 3D ConvNets) is used as the backbone to directly extract features. Zhang Yueyu et al. use the first feature extracted by I3D to obtain the EMD (Earth Mover's Distance) distance describing the distance, use the EMD distance to construct the inter-frame similarity matrix and obtain the similarity between videos through twin neural network training.

[0004] However, for videos with camera shake and continuous motion of the video target, the extracted key frames are relatively blurry. The method of reducing the dimension of the video to an image through key frames is not suitable for manual feature extraction. In addition, for videos with text embedded in the video screen, if the key frames are detected with corner points (such as SIFT), the obtained feature points are likely to be concentrated on the edge of the text and cannot reflect the features of the original video screen. At this time, the only way is to use the pre-trained model of deep learning to extract the mid- and high-level semantic information of the key frames as the features of the image, but this approach is very time-consuming and not suitable for scenes with a large number of videos. Moreover, the multiple key frames of the video contain a lot of redundant information, and the extracted features will also contain a lot of redundancy, which will cause the recognition effect of the convolutional neural network to deteriorate. The method using optical flow has high computational complexity and is not suitable for large-scale short video deduplication scenarios. In addition, the deduplication method based on video frames does not consider the characteristics of the time dimension, resulting in the inability to distinguish repeated videos with different pictures but similar content. Summary of the invention

[0005] In order to solve the above problems existing in the prior art, the present invention provides a short video deduplication method based on deep learning. The technical problem to be solved by the present invention is achieved by the following technical solutions:

[0006] The present invention provides a short video deduplication method based on deep learning, comprising:

[0007] Preprocessing the target video and the query video to be compared to obtain tensor representations of the target video and the query video respectively;

[0008] Using a pre-trained neural network to obtain the spatiotemporal feature vectors of the target video and the query video respectively;

[0009] Constructing a neural network model for video feature similarity discrimination and training the neural network model, the neural network model includes a weight layer, a fusion module and a 2D convolutional neural network module, wherein the weight layer is used to weight the spatiotemporal feature vectors of the target video and the query video respectively to obtain the weighted spatiotemporal feature vectors of the target video and the query video respectively, the fusion module is used to perform feature fusion on the weighted spatiotemporal feature vectors of the target video and the query video to obtain the fused spatiotemporal feature vector; the 2D convolutional neural network module is used to obtain the similarity value of the target video and the query video according to the fused spatiotemporal feature vector;

[0010] Inputting the spatiotemporal feature vectors of the target video and the query video into a trained neural network model to obtain a similarity value between the target video and the query video;

[0011] The similarity value is used to determine whether to delete the target video and the query video.

[0012] In one embodiment of the present invention, the target video and the query video to be compared are preprocessed to obtain tensor representations of the target video and the query video respectively, including:

[0013] Uniformly extracting multiple frames of images from the target video in a time dimension, removing redundant images that are adjacent and have similar structures from the multiple frames of images, and obtaining multiple frames of images after the redundant images are removed;

[0014] Deleting images whose blurriness exceeds a set blurriness threshold value from the multiple frames of images, and obtaining multiple frames of remaining images after blurry frames are removed;

[0015] Obtaining a tensor representation of a target video according to the plurality of frames of residual images;

[0016] The query video is preprocessed according to the preprocessing process of the target video to obtain a tensor representation of the query video.

[0017] In one embodiment of the present invention, removing redundant images that are adjacent and have similar structures in the multiple frames of images includes:

[0018] The brightness similarity, contrast similarity and structural similarity of adjacent frame images are obtained respectively:

[0019]

[0020] Among them, l(x,y), c(x,y) and s(x,y) represent the similarity between the brightness, contrast and structure of two adjacent grayscale images in the current window, x and y are the respective pixel sets in the current N×N window after the adjacent frame images are grayed, μ x is the mean x-axis coordinate of all pixels in the current window, μ y is the mean of the y-axis coordinates of all pixels in the current window, is the variance of the x-axis coordinates of all pixels in the current window, is the variance of the y-axis coordinates of all pixels in the current window, σ xy is the covariance of the x-axis coordinates and y-axis coordinates of all pixels in the current window, c1, c2 and c3 are constants;

[0021] Get the total similarity value between adjacent frame images:

[0022] SSIM(x,y)=[l(x,y) α c(x,y) β s(x,y) γ ],

[0023] Among them, α, β, and γ are set constants;

[0024] The total similarity value between adjacent frame images is compared with a set similarity threshold. If the total similarity value is greater than the set similarity threshold, one frame of the adjacent frame images is randomly deleted, and adjacent frame images of the target video multi-frame images are compared in pairs until the total similarity values ​​between all remaining adjacent frame images are less than the similarity threshold.

[0025] In one embodiment of the present invention, the spatiotemporal feature vectors of the target video and the query video are obtained respectively using a pre-trained neural network, including:

[0026] The preprocessed multiple frames of the target video or the query video are respectively input into the pre-trained VisionTransformer network to obtain the spatial feature vector of each image, and then the spatial feature vector is fused with the position encoding and input into the temporal feature encoder of the pre-trained Video transformer network to obtain the spatiotemporal feature vector of the target video or the query video.

[0027] In one embodiment of the present invention, the 2D convolutional neural network includes a copy padding layer, a first two-dimensional convolutional layer, a copy padding layer, a second two-dimensional convolutional layer, a copy padding layer, a third two-dimensional convolutional layer and a fourth two-dimensional convolutional layer connected in sequence, wherein:

[0028] Each replication filling layer is used to fill a circle of values ​​around the input matrix;

[0029] The activation functions of the first two-dimensional convolutional layer, the second two-dimensional convolutional layer and the third two-dimensional convolutional layer are relu, and maximum pooling is adopted. The fourth two-dimensional convolutional layer outputs a similarity matrix using a softmax layer.

[0030] In one embodiment of the present invention, training the neural network model includes:

[0031] Constructing a training data set, wherein the training data set includes a plurality of video triplets, each video triplet includes an anchor video, a positive video having content similar to the anchor video, and a negative video having content dissimilar to the anchor video;

[0032] Preprocessing each video in the video triplet to obtain a tensor representation of each video after preprocessing;

[0033] Using a pre-trained neural network, respectively obtain the spatiotemporal feature vectors of the anchor video, the positive video, and the negative video;

[0034] Weighting the spatiotemporal feature vectors of the anchor video, the positive video, and the negative video respectively to obtain weighted spatiotemporal feature vectors of the anchor video, the positive video, and the negative video;

[0035] The weighted spatiotemporal feature vectors of the anchor video and the positive video are fused to obtain a first joint matrix, and the weighted spatiotemporal feature vectors of the anchor video and the negative video are fused to obtain a second joint matrix;

[0036] The first joint matrix is ​​input into the 2D convolutional neural network to obtain the similarity value between the anchor video and the positive video, the second joint matrix is ​​input into the 2D convolutional neural network to obtain the similarity value between the anchor video and the negative video, and the loss function L=Sim a,n -Sim a,p Update the parameters of the weight layer and the 2D convolutional neural network module, Sim a,n Represents the similarity value between the anchor video and the negative video, Sim a,p Represents the similarity value between the anchor video and the positive video;

[0037] The weight layer and the 2D convolutional neural network module are iteratively trained and updated using the video triplets in the training data set to obtain a trained neural network model.

[0038] In one embodiment of the present invention, inputting the spatiotemporal feature vectors of the target video and the query video into a trained neural network model to obtain a similarity value between the target video and the query video includes:

[0039] The spatiotemporal feature vectors of the target video and the query video are weighted by using the weight layer to obtain the weighted spatiotemporal feature vector of the target video and the weighted spatiotemporal feature vector of the query video and perform feature fusion;

[0040] Put the fused features into the trained 2D convolutional neural network, and output a similarity matrix between the target video and the query video;

[0041] The similarity matrix is ​​subjected to chamfer similarity calculation to obtain a similarity value between the target video and the query video.

[0042] In one embodiment of the present invention, using the obtained similarity value to determine whether to delete the target video and the query video includes:

[0043] After the trained 2D convolutional neural network obtains the similarity value of the target video and the query video, it is compared with the set similarity threshold. When the similarity value exceeds the similarity threshold, the video with a shorter duration or lower resolution between the target video and the query video is deleted.

[0044] Another aspect of the present invention provides a storage medium, in which a computer program is stored, and the computer program is used to execute the steps of the short video deduplication method based on deep learning in any one of the above embodiments.

[0045] Another aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and when the processor calls the computer program in the memory, it implements the steps of the short video deduplication method based on deep learning as described in any of the above embodiments.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1. The short video deduplication method based on deep learning of the present invention uses a video understanding algorithm to calculate the video repetition in the case of short video deduplication, storage and management, solving the problem of short video detection for repeated video content. The method has a novel framework, fast algorithm convergence, high efficiency, and does not require additional calculation of optical flow. Especially for semantic deduplication with similar content, it can calculate the theme-based video feature library relatively quickly, improving the efficiency of short video deduplication.

[0048] 2. The short video deduplication method of the present invention uses a pre-trained model in the process of spatiotemporal feature extraction, does not need to be trained from scratch, and can use a public similar video retrieval dataset, without the need for additional manual construction of a training dataset.

[0049] 3. The short video deduplication method of the present invention is robust and has strong generalization, and can achieve good deduplication effects for different types of videos in the short video industry.

[0050] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is a flow chart of a short video deduplication method based on deep learning provided by an embodiment of the present invention;

[0052] Figure 2 It is a schematic diagram of the working process of an encoder in a Vision Transformer network provided by an embodiment of the present invention;

[0053] Figure 3 Schematic diagram of a spatiotemporal feature vector extraction process provided by an embodiment of the present invention;

[0054] Figure 4 is a structural diagram of a weight layer provided by an embodiment of the present invention;

[0055] Figure 5 is a schematic diagram of the structure of a 2D convolutional neural network provided by an embodiment of the present invention;

[0056] Figure 6 It is a schematic diagram of a model training process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0057] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose of the invention, the following is a detailed description of a short video deduplication method based on deep learning proposed in accordance with the present invention in combination with the accompanying drawings and specific implementation methods.

[0058] The above and other technical contents, features and effects of the present invention are clearly presented in the following detailed description of the specific implementation modes in conjunction with the accompanying drawings. Through the description of the specific implementation modes, the technical means and effects adopted by the present invention to achieve the predetermined purpose can be more deeply and specifically understood. However, the attached drawings are only for reference and explanation purposes and are not used to limit the technical solutions of the present invention.

[0059] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants are intended to cover non-exclusive inclusion, so that an article or device including a series of elements includes not only those elements, but also other elements that are not explicitly listed. In the absence of more restrictions, the elements defined by the statement "including one..." do not exclude the existence of other identical elements in the article or device including the elements.

[0060] See also Figure 1 , Figure 1 : is a flowchart of a short video deduplication method based on deep learning provided by an embodiment of the present invention. The method comprises the following steps:

[0061] S1: Preprocess the target video and the query video to be compared, and obtain tensor representations of the target video and the query video respectively.

[0062] Step S1 of this embodiment specifically includes the following steps:

[0063] S1.1: Extract multiple frames of images uniformly from the target video in the time dimension according to the video duration, and remove adjacent redundant images with similar structures from the multiple frames according to the structure similarity index measure (SSIM), to obtain multiple frames of images after removing redundant images. A similarity threshold s can be set, and a similarity value is calculated for each pair of adjacent frames in the extracted time dimension through SSIM. If the SSIM value is greater than the similarity threshold s, one of the two frames is randomly deleted until the SSIM values ​​between all remaining adjacent frames are less than the similarity threshold s.

[0064] Specifically, the adjacent frames that need to be calculated for similarity are first converted into grayscale images. In each calculation, an N×N window is selected from the image, and then the window is continuously slid to calculate the similarity, and finally the average value is taken as the SSIM value of the entire image.

[0065] The SSIM metric is calculated as follows:

[0066] The brightness similarity, contrast similarity and structural similarity of adjacent frame images are obtained respectively:

[0067]

[0068] Among them, l(x,y), c(x,y) and s(x,y) represent the similarity between the brightness (luminance), contrast (constract) and structure (structure) of the two grayscale images in the current window, x and y are the respective pixel sets of the two images to be compared in the current N×N window after grayscale conversion, μ x is the mean x-axis coordinate of all pixels in the current window, μ y is the mean of the y-axis coordinates of all pixels in the current window, is the variance of the x-axis coordinates of all pixels in the current window, is the variance of the y-axis coordinates of all pixels in the current window, σ xy is the covariance of the x-axis coordinates and y-axis coordinates of all pixels in the current window, c1=(k1L) 2 ,c2=(k2L) 2 are two constants, c3=c2 / 2 avoids division by zero, L is the pixel value range, generally 255, k1=0.01, k2=0.03 are the default values. The above three formulas respectively calculate the similarity between the brightness, contrast and structure of the two grayscale images in the current window.

[0069] Then, the total SSIM of the pictures of adjacent frames is obtained:

[0070] SSIM(x,y)=[l(x,y) α c(x,y) β s(x,y) γ ],

[0071] Among them, α, β, γ are set to 1, and c3=c2 / 2, we can get:

[0072]

[0073] If the calculated SSIM value is greater than the set threshold s, one of the two frames is randomly deleted, and the adjacent frame images captured from each video are compared pairwise until the SSIM values ​​between all remaining adjacent frames are less than the threshold s.

[0074] S1.2: Delete the images whose blurriness exceeds the set blurriness threshold in the multiple frames of images after the redundant images are removed, and obtain the multiple frames of remaining images after the blurred frames are removed.

[0075] Since the image features extracted from blurred images are inaccurate, which will affect the subsequent spatiotemporal feature extraction of video clips, this embodiment also needs to detect and remove blurred images. Specifically, all the remaining frames of the target video obtained in step S1.1 are copied first, and then the images are grayed and convolved with a step length of 1 using the Laplace operator to obtain a new matrix, and the variance of the new matrix is ​​calculated as the blurriness, and the blurriness value is compared with a preset blurriness threshold. If the blurriness value exceeds the blurriness threshold, the frame image is deleted from the original image set.

[0076] S1.3: Obtain a tensor representation of the target video based on the multiple frames of residual images after removing the blurred frames.

[0077] Specifically, the short sides of the remaining frame images are scaled to 256, and then a 224×224 image is cropped at the center of the original image. These images are then collected to obtain the tensor representation of the target video, which is represented as follows: P1∈R K×224×224×3 , where P1 represents the tensor representation of the target video, K represents the number of remaining pictures, and 3 represents the number of picture channels.

[0078] S1.4: Preprocess the query video according to the preprocessing process of the target video to obtain a tensor representation of the query video.

[0079] Similarly, according to steps S1.1-S1.3, multiple frames of images are evenly extracted from the query video in the time dimension according to the video duration, and adjacent and structurally similar redundant images in the multiple frames are removed according to the structural similarity index to obtain multiple frames of images after removing redundant images; images whose blurriness exceeds a set blurriness threshold in the multiple frames after removing redundant images are deleted to obtain multiple frames of images after removing blurred frames; and based on the multiple frames of remaining images after removing blurred frames, a tensor representation of the query video is obtained.

[0080] S2: Using a pre-trained neural network, respectively obtain the spatiotemporal feature vectors of the target video and the query video.

[0081] Each frame of each video (target video or query video) is input into the pre-trained VisionTransformer network to obtain the spatial feature vector of each image, and then these spatial feature vectors are fused with position encoding and input into the temporal feature encoder of the pre-trained Video transformer network to obtain the spatiotemporal feature vector of each video (target video or query video).

[0082] Specifically, step S2 of this embodiment includes the following steps:

[0083] S2.1: See Figure 2 , Figure 2 Schematic diagram of the working process of the encoder in the Vision Transformer network provided by an embodiment of the present invention. A frame of the target video is input into the pre-trained Vision Transformer network, and after 6 steps such as Figure 2 After the encoder shown, the spatial features of the current image are obtained, and then the spatial feature vector of each frame image in the target video can be obtained. Similarly, each frame image in the query video is input into the pre-trained Vision Transformer network, and the spatial feature vector of each frame image in the query video can also be obtained.

[0084] S2.2: Input the spatial feature vectors of all images in the target video into the temporal feature extractor of the pre-trained VideoTransformer network in batches to obtain the spatiotemporal features between each batch of images of the target video, combine all the spatiotemporal features of the target video through the concat function to form the spatiotemporal features of the entire target video and perform l2-norm normalization on the spatiotemporal features.

[0085] See also Figure 3 , Figure 3It is a schematic diagram of a spatiotemporal feature vector extraction process provided by an embodiment of the present invention. Wherein, Spatial attention-based Encoder represents a temporal encoder based on self-attention, and Temporal attention-based Encoder represents a spatial encoder based on self-attention. In this embodiment, the video, such as a frame of the target video, is first divided into many patches (blocks) of size 16×16. Since the image after preprocessing in step S1 becomes 224×224×3, 3 represents the three channels of the RGB image, a total of 196 (224×224 / (16×16)) patches can be obtained here, and each patch is expanded to obtain some vectors of length 16×16×3=768. PE is a position code, which indicates the relative position of each divided patch in the picture, for example, the fourth block (3, 4) of the third row is mapped to a vector of length 768. CLS is a classification head, which is a randomly generated vector of length 768. M and N correspond to the number of self-attention-based temporal encoders and self-attention-based spatial encoders, respectively. In this embodiment, M and N are both 6.

[0086] S3: Construct a neural network model for video feature similarity discrimination and train the neural network model, wherein the neural network model includes a weight layer, a fusion module and a 2D convolutional neural network module, wherein the weight layer is used to weight the spatiotemporal feature vectors of the target video and the query video respectively to obtain the weighted spatiotemporal feature vectors of the target video and the query video respectively, the fusion module is used to perform feature fusion on the weighted spatiotemporal feature vectors of the target video and the query video to obtain the fused spatiotemporal feature vector; the 2D convolutional neural network module is used to obtain the similarity value of the target video and the query video according to the fused spatiotemporal feature vector.

[0087] Furthermore, if Figure 5 As shown, the 2D convolutional neural network includes a copy and fill layer, a first two-dimensional convolutional layer, a copy and fill layer, a second two-dimensional convolutional layer, a copy and fill layer, a third two-dimensional convolutional layer and a fourth two-dimensional convolutional layer connected in sequence, wherein each copy and fill layer is used to fill a circle of values ​​around the input matrix; the activation function of the first two-dimensional convolutional layer, the second two-dimensional convolutional layer and the third two-dimensional convolutional layer is relu, and maximum pooling is used, and the fourth two-dimensional convolutional layer uses a softmax layer to output a similarity matrix.

[0088] Specifically, the weight layer first weights the different spatiotemporal feature vectors of the target video or the query video. The weight layer uses a fully connected neural network to map the features of different video clips to the corresponding number of outputs, and then passes through softmax to multiply the weights obtained by the corresponding video clip features. Figure 4 , Figure 4 It is a structural diagram of a weight layer provided by an embodiment of the present invention.

[0089] The specific steps for weighting the spatiotemporal features are as follows: first, for the spatiotemporal feature map of the target video or the query video (a matrix of size N×d, where N represents the number of spatiotemporal feature maps and d represents the dimension of the spatiotemporal feature map), the average value of each spatiotemporal feature map is obtained to obtain a vector of length N, which is then input into a two-layer neural network. The first layer uses the relu activation function and the second layer uses the sigmoid activation function. Finally, a vector of length N is obtained as the weight, which is then multiplied with the original spatiotemporal feature map to obtain the weighted spatial feature map.

[0090] Furthermore, if Figure 6 As shown, training the neural network model includes:

[0091] Construct a training data set, wherein the training data set includes multiple video triplets, each video triplet includes an anchor video, a positive video with similar content to the anchor video, and a negative video with dissimilar content to the anchor video. Specifically, in this embodiment, an anchor video is randomly selected from a large number of videos, and the anchor video is edited to a certain extent to serve as a positive video. The editing means may include color transformation (contrast, grayscale, etc.), geometric shape transformation (horizontal, vertical flip, rotation and scaling, etc.), and time transformation (fast forward, slow motion, pause, etc.). It is necessary to ensure that the screen content of at least 4 seconds between the anchor video and the positive video is similar. The negative video is selected from a large number of videos and needs to be completely irrelevant to the anchor video in content.

[0092] Preprocess each video in the video triplet to obtain a tensor representation of each video after preprocessing. Specifically, extract multiple frames of images uniformly according to the video duration in the time dimension for each video, remove adjacent and structurally similar redundant frames in the multiple frames of images according to the Structure Similarity Index Measure (SSIM), delete images whose blurriness exceeds a set blurriness threshold in the images captured from each video, and obtain a tensor representation of the captured images of each video. For the specific processing process, please refer to step S1, which will not be repeated here.

[0093] Subsequently, a pre-trained neural network is used to obtain the spatiotemporal feature vectors of the anchor video, the positive video and the negative video respectively, and the spatiotemporal feature vectors of the anchor video, the positive video and the negative video are weighted respectively to obtain the weighted spatiotemporal feature vectors of the anchor video, the positive video and the negative video. For the specific processing process, please refer to step S2, which will not be repeated here.

[0094] Subsequently, the weighted spatiotemporal feature vectors of the anchor video and the positive video are fused to obtain a first joint matrix, and the weighted spatiotemporal feature vectors of the anchor video and the negative video are fused to obtain a second joint matrix.

[0095] For example, for a positive sample pair (anchor video and positive video), for the feature vectors of the two videos (N1×d, N2×d), the feature vector of the positive video is transposed and multiplied with the feature vector of the anchor video to obtain a joint matrix of N1×N2.

[0096] The first joint matrix is ​​input into the 2D convolutional neural network to obtain a similarity matrix between the anchor video and the positive video, and the second joint matrix is ​​input into the 2D convolutional neural network to obtain a similarity matrix between the anchor video and the negative video; and then chamfer similarity calculation is performed on the output similarity matrix of size N×M:

[0097]

[0098] Among them, S(i,j) represents a value in the i-th row and j-th column of the similarity matrix. The formula is to take the maximum value of each row and take the average value as the final similarity value, and then obtain the similarity value between the anchor video and the positive video, and the similarity value between the anchor video and the negative video.

[0099] Furthermore, using the loss function L = Sim a,n -Sim a,p Update the parameters of the weight layer and the 2D convolutional neural network module, and use the video triples in the training data set to iteratively train and update the weight layer and the 2D convolutional neural network module until the set number of iterations is reached to obtain a trained neural network model.

[0100] S4: Input the spatiotemporal feature vectors of the target video and the query video into a trained neural network model to obtain a similarity value between the target video and the query video.

[0101] Specifically, the weight layer is used to weight the spatiotemporal feature vectors of the target video and the query video respectively, to obtain the weighted spatiotemporal feature vector of the target video and the weighted spatiotemporal feature vector of the query video and perform feature fusion; the fused features are put into the trained 2D convolutional neural network, and the similarity matrix of the target video and the query video is output; the chamfer similarity calculation is performed on the similarity matrix to obtain the similarity value of the target video and the query video.

[0102] In the actual processing, given two videos, the two videos are named query video and target video respectively, and the calculation of feature fusion is: F combine =F query ·(F target ) T , F query is the feature of the query video, F target is the feature of the target video, F combine It is the fused feature. After the video similarity learning network, a similarity is finally generated. Then the roles of the two videos are swapped, that is, the original target video is used as the query video, and the original query video is used as the target video to be re-fused and a similarity value is calculated. Finally, the average of the two similarities is taken as the final similarity value.

[0103] S5: Using the similarity value, determine whether to delete the target video and the query video.

[0104] Specifically, after the trained 2D convolutional neural network obtains the similarity value of the target video and the query video, it is compared with a set threshold. When the similarity value exceeds the threshold, the video with a shorter duration or lower resolution between the target video and the query video is deleted.

[0105] The embodiment of the present invention is a short video deduplication method based on deep learning. It uses a video understanding algorithm to calculate video repetition in the case of short video deduplication, storage and management, and solves the problem of short video detection for repeated video content. The algorithm framework proposed by this method is novel, the algorithm converges quickly, and the efficiency is high, and no additional calculation of optical flow is required. In particular, for semantic deduplication with similar content, it can calculate a theme-based video feature library relatively quickly, thereby improving the efficiency of short video deduplication. The short video deduplication method of the present invention uses a pre-trained model in the process of spatiotemporal feature extraction, does not require training from scratch, and can use a public similar video retrieval data set, without the need for additional manual construction of a training data set.

[0106] Another embodiment of the present invention provides a storage medium, in which a computer program is stored, and the computer program is used to execute the steps of the short video deduplication method based on deep learning described in the above embodiment. Another aspect of the present invention provides an electronic device, including a memory and a processor, in which a computer program is stored, and when the processor calls the computer program in the memory, the steps of the short video deduplication method based on deep learning as described in the above embodiment are implemented. Specifically, the above-mentioned integrated module implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including several instructions for an electronic device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform some steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0107] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. A short video deduplication method based on deep learning, characterized in that: include: Preprocessing the target video and the query video to be compared to obtain tensor representations of the target video and the query video respectively; Using a pre-trained neural network to obtain the spatiotemporal feature vectors of the target video and the query video respectively; Constructing a neural network model for video feature similarity discrimination and training the neural network model, the neural network model includes a weight layer, a fusion module and a 2D convolutional neural network module, wherein the weight layer is used to weight the spatiotemporal feature vectors of the target video and the query video respectively to obtain the weighted spatiotemporal feature vectors of the target video and the query video respectively, the fusion module is used to perform feature fusion on the weighted spatiotemporal feature vectors of the target video and the query video to obtain the fused spatiotemporal feature vector; the 2D convolutional neural network module is used to obtain the similarity value of the target video and the query video according to the fused spatiotemporal feature vector; Inputting the spatiotemporal feature vectors of the target video and the query video into a trained neural network model to obtain a similarity value between the target video and the query video; The similarity value is used to determine whether to delete the target video and the query video, wherein: The spatiotemporal feature vectors of the target video and the query video are obtained respectively using a pre-trained neural network, including: Inputting the preprocessed multiple frames of the target video or the query video into the pre-trained VisionTransformer network to obtain the spatial feature vector of each image, and then inputting the spatial feature vector into the temporal feature encoder of the pre-trained Video transformer network with the fusion position code to obtain the spatiotemporal feature vector of the target video or the query video; Training the neural network model includes: Constructing a training data set, wherein the training data set includes a plurality of video triplets, each video triplet includes an anchor video, a positive video having content similar to the anchor video, and a negative video having content dissimilar to the anchor video; Preprocessing each video in the video triplet to obtain a tensor representation of each video after preprocessing; Using a pre-trained neural network, respectively obtain the spatiotemporal feature vectors of the anchor video, the positive video, and the negative video; Weighting the spatiotemporal feature vectors of the anchor video, the positive video, and the negative video respectively to obtain weighted spatiotemporal feature vectors of the anchor video, the positive video, and the negative video; The weighted spatiotemporal feature vectors of the anchor video and the positive video are fused to obtain a first joint matrix, and the weighted spatiotemporal feature vectors of the anchor video and the negative video are fused to obtain a second joint matrix; The first joint matrix is ​​input into the 2D convolutional neural network to obtain the similarity value between the anchor video and the positive video, the second joint matrix is ​​input into the 2D convolutional neural network to obtain the similarity value between the anchor video and the negative video, and the loss function L=Sim a,n -Sim a,p Update the parameters of the weight layer and the 2D convolutional neural network module, where Sim a,n Represents the similarity value between the anchor video and the negative video, Sim a,p Represents the similarity value between the anchor video and the positive video; The weight layer and the 2D convolutional neural network module are iteratively trained and updated using the video triplets in the training data set to obtain a trained neural network model.

2. The short video deduplication method based on deep learning according to claim 1, characterized in that: The target video and the query video to be compared are preprocessed to obtain tensor representations of the target video and the query video, respectively, including: Uniformly extracting multiple frames of images from the target video in a time dimension, removing redundant images that are adjacent and have similar structures from the multiple frames of images, and obtaining multiple frames of images after the redundant images are removed; Deleting images whose blurriness exceeds a set blurriness threshold value from the multiple frames of images, and obtaining multiple frames of remaining images after blurry frames are removed; Obtaining a tensor representation of a target video according to the plurality of frames of residual images; The query video is preprocessed according to the preprocessing process of the target video to obtain a tensor representation of the query video.

3. The short video deduplication method based on deep learning according to claim 2 is characterized in that: Removing redundant images that are adjacent and have similar structures from the multiple frames of images, including: The brightness similarity, contrast similarity and structural similarity of adjacent frame images are obtained respectively: Among them, l(x,y), c(x,y) and s(x,y) represent the similarity between the brightness, contrast and structure of two adjacent grayscale images in the current window, x and y are the respective pixel sets in the current N×N window after the adjacent frame images are grayed, μ x is the mean x-axis coordinate of all pixels in the current window, μ y is the mean of the y-axis coordinates of all pixels in the current window, is the variance of the x-axis coordinates of all pixels in the current window, is the variance of the y-axis coordinates of all pixels in the current window, σ xy is the covariance of the x-axis coordinates and y-axis coordinates of all pixels in the current window, c1, c2 and c3 are constants; Get the total similarity value between adjacent frame images: SSIM(x,y)=[l(x,y) α ·c(x,y) β ·s(x,y) γ ], Among them, α, β, and γ are set constants; The total similarity value between adjacent frame images is compared with a set similarity threshold. If the total similarity value is greater than the set similarity threshold, one frame of the adjacent frame images is randomly deleted, and adjacent frame images of the target video multi-frame images are compared in pairs until the total similarity values ​​between all remaining adjacent frame images are less than the similarity threshold.

4. The short video deduplication method based on deep learning according to claim 1, characterized in that: The 2D convolutional neural network includes a copy padding layer, a first two-dimensional convolutional layer, a copy padding layer, a second two-dimensional convolutional layer, a copy padding layer, a third two-dimensional convolutional layer and a fourth two-dimensional convolutional layer connected in sequence, wherein: Each replication filling layer is used to fill a circle of values ​​around the input matrix; The activation functions of the first two-dimensional convolutional layer, the second two-dimensional convolutional layer and the third two-dimensional convolutional layer are relu, and maximum pooling is adopted. The fourth two-dimensional convolutional layer outputs a similarity matrix using a softmax layer.

5. The short video deduplication method based on deep learning according to claim 1, characterized in that: Inputting the spatiotemporal feature vectors of the target video and the query video into a trained neural network model to obtain a similarity value between the target video and the query video, comprising: The spatiotemporal feature vectors of the target video and the query video are weighted by using the weight layer to obtain the weighted spatiotemporal feature vector of the target video and the weighted spatiotemporal feature vector of the query video and perform feature fusion; Put the fused features into the trained 2D convolutional neural network, and output a similarity matrix between the target video and the query video; The similarity matrix is ​​subjected to chamfer similarity calculation to obtain a similarity value between the target video and the query video.

6. The short video deduplication method based on deep learning according to claim 1, characterized in that: Using the obtained similarity value to determine whether to delete the target video and the inquiry video includes: After the trained 2D convolutional neural network obtains the similarity value of the target video and the query video, it is compared with the set threshold. When the similarity value exceeds the threshold, the video with a shorter duration or lower resolution between the target video and the query video is deleted.

7. A storage medium, characterized in that: The storage medium stores a computer program, which is used to execute the steps of the short video deduplication method based on deep learning as described in any one of claims 1 to 6.

8. An electronic device, characterized in that: It includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor calls the computer program in the memory, the steps of the short video deduplication method based on deep learning as described in any one of claims 1 to 6 are implemented.

Images