Video retrieval method, storage medium and device

By calculating the mutual power spectrum and phase correlation method of the video feature matrix, the similarity and offset of the video frame are determined, and the problems of difficulty in judging video similarity and low computing efficiency in the prior art are solved, and efficient video retrieval is achieved.

CN113704551BActive Publication Date: 2024-11-15GUANGZHOU HUYA TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110975922.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-08-24
Publication Date
2024-11-15
Estimated Expiration
2041-08-24

AI Technical Summary

Technical Problem

The existing video retrieval methods cannot determine the specific repetitive clips when judging video similarity, and the sliding window method has low calculation efficiency, making it difficult to meet the requirements of the video creation platform.

Method used

By obtaining the feature matrix of the video to be retrieved and the candidate video, the similarity and offset are calculated using the mutual power spectrum and phase correlation method, the position of similar video frames in timing is determined, thereby improving the video retrieval efficiency.

Benefits of technology

The efficiency of video retrieval is improved, without making judgments on each video frame, accurately determining the location of similar video frames, and meeting the efficient retrieval needs of the video creation platform.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113704551B_ABST
    Figure CN113704551B_ABST
Patent Text Reader

Abstract

The present specification provides a video retrieval method, storage medium and device. In the method, based on the cross power spectrum between the feature matrix of the video to be retrieved and the feature matrix of the candidate video obtained from the video library, the similarity between the two videos and the offset in time sequence of the similar video frames in the two videos are obtained, so that when the similarity exceeds a threshold, the position of the similar video frames in the two videos can be determined based on the offset. In this way, by using the phase correlation method to match the video to be retrieved with the video in the video library, there is no need to make a judgment on each video frame, thereby improving the efficiency of video retrieval.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a video retrieval method, storage medium and device. Background Art

[0002] In order to protect the copyright of the creator, when adding a new video to the video library of the video platform, the video platform needs to retrieve several videos with similar content to the new video from the existing videos in the video library. The related technology generally uses MFH (Multiple Feature Hashing), STH (Self-Taught Hashing), sliding window method, etc. to search for videos. However, MFH method and STH extract a corresponding feature for each video, and judge whether two videos are similar based on the feature, but cannot determine which specific clips are repeated; and the sliding window method requires a large number of sliding judgments on each frame of the video, which has a relatively low computational efficiency and is difficult to meet the requirements of the video creation platform. Summary of the invention

[0003] In order to overcome the problems existing in the related art, this specification provides a video retrieval method, storage medium and device.

[0004] According to a first aspect of an embodiment of this specification, a video retrieval method is provided, comprising:

[0005] Obtain the feature matrix of the video to be retrieved;

[0006] Based on the cross power spectrum between the feature matrix of the video to be retrieved and the feature matrix of the candidate video, the similarity and offset between the video to be retrieved and the candidate video are obtained; the candidate video is obtained from a video library; the offset represents the temporal offset of similar video frames in the video to be retrieved and the candidate video;

[0007] If the similarity exceeds a threshold, a first position of the similar video frame in the video to be retrieved and a second position of the similar video frame in the candidate video are determined based on the offset.

[0008] In some examples, the candidate videos are obtained from a video library based on the video fingerprint of the video to be retrieved.

[0009] In some examples, the video fingerprint of the video to be retrieved / candidate video is composed of singular values ​​obtained by performing singular value decomposition on the feature matrix of the video to be retrieved / candidate video.

[0010] In some examples, the video fingerprint of the video to be retrieved / candidate video is obtained based on the following method:

[0011] Performing singular value decomposition on the feature matrix of the video to be retrieved / candidate video, and forming an initial vector with the obtained singular values;

[0012] A preset dimension in the initial vector is intercepted as a target vector, and the target vector is determined as a video fingerprint of the video to be retrieved / candidate video.

[0013] In some examples, obtaining the similarity and offset between the video to be retrieved and the candidate video based on the cross power spectrum between the feature matrix of the video to be retrieved and the feature matrix of the candidate video includes:

[0014] Performing inverse Fourier transform on the cross power spectrum between the feature matrix of the video to be retrieved and the feature matrix of the candidate video to obtain an impulse function;

[0015] A peak point is obtained based on the impulse function, and the similarity and offset between the video to be retrieved and the candidate video are obtained according to the coordinates of the peak point.

[0016] In some examples, before obtaining the similarity and offset between the video to be retrieved and the candidate video based on the cross power spectrum between the feature matrix of the video to be retrieved and the feature matrix of the candidate video, the method includes:

[0017] If the number of frames of the video to be retrieved is not equal to the number of frames of the candidate video, at least one of the feature matrix of the candidate video and the feature matrix of the video to be retrieved is segmented so that the feature matrix of the candidate video and the feature matrix of the video to be retrieved after segmentation have the same number of rows and columns.

[0018] In some examples, if the similarity is greater than or equal to a first threshold, and the offset is equal to zero, it is determined that the similar video frames include all video frames of the video to be retrieved and the candidate video;

[0019] If the similarity is less than the first threshold and greater than the second threshold, and the offset is equal to zero, the video to be retrieved and the candidate video are segmented, and the similarity and offset between the segmented videos are redetermined to determine the position of the similar video frame according to the redetermined similarity and offset; wherein the first threshold is greater than the second threshold.

[0020] In some examples, the method further comprises:

[0021] Determine a first position of the similar video frame in the video to be retrieved and a second position of the similar video frame in the candidate video as an initial matching result;

[0022] Segmenting a first matrix corresponding to the first position from the feature matrix of the video to be retrieved, and segmenting a second matrix corresponding to the second position from the feature matrix of the candidate video;

[0023] Based on the cross power spectrum between the first matrix and the second matrix, obtaining the target similarity and target offset between the first matrix and the second matrix;

[0024] If the target similarity is greater than or equal to the first threshold, and the target offset is equal to zero, it is determined that the initial matching result is correct.

[0025] According to a second aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, any method in the embodiments of the specification is implemented.

[0026] According to a third aspect of the embodiments of this specification, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any one of the methods in the embodiments of the specification is implemented.

[0027] The technical solutions provided by the embodiments of this specification may have the following beneficial effects:

[0028] In the embodiments of this specification, a video retrieval method, storage medium and device are disclosed. In this method, based on the cross power spectrum between the feature matrix of the video to be retrieved and the feature matrix of the candidate video obtained from the video library, the similarity between the two videos and the offset in time sequence of the similar video frames in the two videos are obtained, so that when the similarity exceeds a threshold, the position of the similar video frames in the two videos can be determined based on the offset. In this way, by using the phase correlation method to match the video to be retrieved with the video in the video library, there is no need to make a judgment on each video frame, thereby improving the efficiency of video retrieval.

[0029] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present specification. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the specification and, together with the description, serve to explain the principles of the specification.

[0031] Figure 1 is a flow chart of a video retrieval method according to an exemplary embodiment of the present specification;

[0032] Figure 2is a schematic diagram of a video retrieval system according to an exemplary embodiment of the present specification;

[0033] Figure 3 It is a hardware structure diagram of a computer device where a video retrieval device is located according to an exemplary embodiment of this specification;

[0034] Figure 4 It is a block diagram of a video retrieval device according to an exemplary embodiment of the present specification. DETAILED DESCRIPTION

[0035] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this specification. Instead, they are merely examples of devices and methods consistent with some aspects of this specification as detailed in the appended claims.

[0036] The terms used in this specification are for the purpose of describing specific embodiments only and are not intended to limit this specification. The singular forms "a", "the" and "the" used in this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0037] It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various information, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of this specification, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0038] With the development of multimedia technology, video creation platforms have become a very popular gathering place for Internet users, and a considerable number of videos flow into the video library of video creation platforms every day. In order to protect the copyright of creators, when a new video needs to enter the video library, the platform needs to retrieve all videos in the video library to retrieve several videos similar to the content of the new video, and hand them over to the system for subsequent inspection, so as to reduce the situation where some videos stolen by some users are displayed on the platform. In the related art, MFH (Multiple Feature Hashing), STH (Self-Taught Hashing), sliding window method, etc. are generally used to retrieve videos. However, MFH and STH extract a corresponding feature for each video, and judge whether two videos are similar based on the features, but cannot determine which specific segments are repeated; and the sliding window method requires a large number of sliding judgments on each frame of the video, and the calculation efficiency is relatively low, which is difficult to meet the requirements of the video creation platform. Based on this, the embodiment of this specification provides a video retrieval solution to solve the above problems.

[0039] Next, the embodiments of this specification are described in detail.

[0040] like Figure 1 As shown, Figure 1 is a flowchart of a video retrieval method according to an exemplary embodiment of the present specification, the method comprising:

[0041] In step 101, a feature matrix of the video to be retrieved is obtained;

[0042] The solution of this embodiment can be applied to a video platform, and the video to be retrieved in this step can be a video uploaded to the video platform by a publisher, or a video downloaded from other websites by an administrator of the video platform. The type of the video to be retrieved can be a movie, animation, short video, etc., which is not limited in this embodiment.

[0043] The feature matrix mentioned in this step can be aggregated by the frame-level features of the video. Frame-level features refer to the features extracted from each frame of the video. In an optional embodiment, the frame-level features of the video can be obtained by mapping each frame of the video using an image hash algorithm. The principle of the image hash algorithm is to map the input image into a short digital sequence, and use the digital sequence to represent the image itself, thereby reducing the image storage cost and computational complexity. Specifically, for a frame of video, it can be divided into several image blocks of the same size, and then orthogonal transforms such as DFT (Discrete Fourier Transform), DCT (Discrete Cosine Transform), etc. are used to extract the hash value of each image block in the transform domain, and then the hash values ​​of all image blocks are arranged into a vector, which is the frame-level feature of the video frame. Of course, in other embodiments, other types of image hash algorithms, or histograms, SIFT (Scale-invariant feature transform), deep learning algorithms, etc. may be used to generate frame-level features of the video, and this manual does not limit this.

[0044] After obtaining the frame-level features of all video frames, they can be arranged in time sequence to obtain the feature matrix of the video. For example, a video consists of T video frames, each video frame is divided into D image blocks, that is, the frame-level feature of each video frame is a D-dimensional vector, then the feature matrix of the video can be a matrix of size T×D, where the value of the i-th row and j-th column is the hash value corresponding to the j-th image block in the i-th video frame.

[0045] In addition, since the frame rates of the videos flowing into the video platform are not necessarily the same, this will affect the accuracy during the subsequent timing alignment. Based on this, before generating the feature matrix of the video, the frame rate of the video can be fixed to a specified value. This step can be achieved by transcoding the original video using tools such as OpenCV or FFmpeg. Of course, in other embodiments, other methods can also be used according to the needs of specific scenarios, and this specification does not limit this.

[0046] In step 102, based on the cross power spectrum between the feature matrix of the video to be retrieved and the feature matrix of the candidate video, the similarity and offset between the video to be retrieved and the candidate video are obtained; the candidate video is obtained from a video library; the offset represents the temporal offset of similar video frames in the video to be retrieved and the candidate video;

[0047] The candidate video mentioned in this step is a video obtained from a video library. The essence of the video library is a multimedia database, which can be a video platform for managing video data and providing a software system based on content retrieval for users. In practical applications, the number of videos in the video library is very large. If the video to be retrieved and all the videos in the video library are matched one by one, the retrieval efficiency is obviously low. Therefore, in some examples, the candidate video can be obtained from the video library based on the video fingerprint of the video to be retrieved. The video fingerprint is a string of fingerprint characters that can uniquely identify the current video based on the video fingerprint technology, which can be considered as a unique feature vector that characterizes the current video. Based on the video fingerprint of the video to be retrieved, a method for comparing vector similarity can be used, such as Faiss (a library for clustering and similarity search) algorithm, Hamming distance, etc., to filter out candidate videos from the video library. The screening criterion can be that the similarity between the video fingerprint of the candidate video and the video fingerprint of the video to be retrieved exceeds a preset value, or the video fingerprint of each video in the video library and the video fingerprint of the video to be retrieved are sorted by similarity, and the candidate video is a video sorted before the preset position. Through this screening, a series of videos similar to the video to be retrieved can be quickly retrieved, thereby reducing the workload and improving retrieval efficiency.

[0048] The video fingerprint of the video to be retrieved or the candidate video can also be obtained by reducing the dimension of the feature matrix of the video to be retrieved or the candidate video. Specifically, it can be implemented by using data dimension reduction algorithms such as PCA (Principal Component Analysis) and NMF (Nonnegative Matrix Factorization). In an optional embodiment, the video fingerprint of the video to be retrieved / candidate video can be composed of singular values ​​obtained by performing singular value decomposition on the feature matrix of the video to be retrieved / candidate video. Singular Value Decomposition (SVD) is also a data dimension reduction algorithm, which can decompose an m×n matrix A into the following form:

[0049] A=UΣV T

[0050] Among them, U and V are both unit orthogonal matrices, U is called the left singular matrix, V is called the right singular matrix, and Σ is the singular value matrix. The singular value matrix has values ​​only on the main diagonal, these values ​​are called singular values, and the rest are 0. The singular value can be considered as the representative value of the original matrix, that is, the singular value can well express the information of the original matrix. After performing SVD on the feature matrix of the video, a vector composed of the singular values ​​in the singular value matrix is ​​obtained, and the vector is used as the video fingerprint. After experimental summary, using SVD to generate fixed-dimensional video fingerprints can effectively improve the detection rate of video retrieval compared to other data dimensionality reduction algorithms. Moreover, SVD can act directly on the feature matrix without additional processing, which is conducive to reducing the burden on the platform when performing task processing.

[0051] It should be noted that, since the number of frames corresponding to the video to be retrieved and the video in the video library may be different, that is, the length in the time dimension is not fixed, in order to facilitate the subsequent use of screening candidate videos, the first N dimensions of the vector composed of singular values ​​can be intercepted as the video fingerprint, where N is a fixed value. In this way, the video fingerprint of each video is a vector of fixed length, so that given a retrieval vector (video fingerprint of the video to be retrieved), it is possible to quickly find several most similar vectors in the library (video fingerprints of candidate videos).

[0052] In addition, in some examples, the video fingerprints and feature matrices of candidate videos can be pre-stored in a database. In this way, when the video platform performs retrieval processing on the video to be retrieved, it can obtain the video fingerprints and feature matrices of the candidate videos from the database without having to calculate them each time, thereby achieving data reuse and improving the efficiency of video retrieval.

[0053] After obtaining the candidate video, the similarity and offset between the two videos can be obtained based on the cross-power spectrum between the feature matrix of the video to be retrieved and the feature matrix of the candidate video. The cross-power spectrum is a method of describing the degree of correlation between two different signals in the frequency domain. The cross-power spectrum between the feature matrix of the video to be retrieved and the feature matrix of the candidate video can be calculated by performing Fourier transform on the two feature matrices respectively to convert the two feature matrices into the frequency domain. The Fourier transform has the following characteristics: the translational motion of the signal in the time domain can be expressed by the phase transformation in the frequency domain. In the related art, this characteristic is generally applied to achieve image registration, and this registration method is called the phase correlation method. In the present embodiment, the cross-power spectrum between the feature matrix of the video to be retrieved and the feature matrix of the candidate video is used to characterize the phase shift between the two videos, thereby obtaining the similarity and offset between the two videos. That is to say, the present embodiment can be considered to use the phase correlation method to match the two videos.

[0054] Specifically, the similarity and offset between the video to be retrieved and the candidate video can be obtained by obtaining the inverse Fourier transform of the mutual power spectrum. By performing an inverse Fourier transform on the mutual power spectrum between the feature matrix of the video to be retrieved and the feature matrix of the candidate video, an impulse function (also called a Dirac function) can be obtained. By determining the coordinates of the peak point in this impulse function, the similarity and offset between the two videos can be obtained. The offset characterizes the temporal offset of similar video frames in the two videos. The similar video frames refer to video frames with the same or similar image content. The image content here includes the color, texture, shape of objects in the image, the scene, the behavior of objects in the image, etc. Therefore, similar video frames in two videos can be considered as repeated segments in the two videos.

[0055] In an optional embodiment, this step can be implemented using the phaseCorrelate function. The phaseCorrelate function is a phase correlation method provided by OpenCV. Compared with other phase correlation methods, this function adds sub-pixel precision positioning, so that more accurate coordinates can be obtained. The feature matrix of the video to be retrieved and the feature matrix of the candidate video are used as inputs of the phaseCorrelate function. The phaseCorrelate function returns the normalized maximum response value and the relative displacement corresponding to this maximum response value, wherein the maximum response value is the response value corresponding to the sub-pixel precision position coordinates of the peak point, which represents the similarity between the feature matrices of the two videos, that is, the similarity mentioned in this step; the relative displacement is the sub-pixel precision position coordinates of the peak point obtained by weighted average calculation based on the response value of the surrounding area of ​​the peak point as the weight of the coordinate position in the surrounding area of ​​the peak point, which represents the displacement required for the feature matrices of the two videos to reach the similarity corresponding to the maximum response value, that is, the offset mentioned in this step.

[0056] In addition, if the number of frames of the video to be retrieved and the candidate video are not equal, the feature matrices of the two videos have different numbers of rows and columns, and the phase correlation method may be used for matching. Based on this, in some examples, before performing the matching of this step, at least one of the feature matrix of the video to be retrieved and the feature matrix of the candidate video can be segmented so that the feature matrix of the candidate video after segmentation has the same number of rows and columns as the feature matrix of the candidate video. For example, the number of frames of the video to be retrieved is T1, and the number of frames of the candidate video is T2. If T1 is less than T2, the feature matrix T2×D of the candidate video is segmented based on T1 into at least two sub-matrices, and the size of each sub-matrix is ​​T1×D; if T1 is greater than T2, the feature matrix T1×D of the video to be retrieved is segmented based on T2 into at least two sub-matrices, and the size of each sub-matrix is ​​T2×D. For a longer video, since only the features of the length of the shorter video can be matched at a time, it is necessary to segment the longer video multiple times along the time dimension. When the remaining part after segmentation does not meet the length of the shorter video, the longer video can be segmented in reverse order along the time dimension. For example, if the video to be retrieved has 15 frames and the candidate video has 9 frames, the feature matrix of the video to be retrieved is segmented into sub-matrices corresponding to the first to ninth frames and sub-matrices corresponding to the seventh to fifteenth frames based on the number of frames of the candidate video. In this way, each sub-matrix obtained by segmenting the feature matrix of the candidate video has the same number of rows and columns as the feature matrix of the video to be retrieved. In this way, when using the phase correlation method to match two videos, the unsegmented feature matrix and the segmented sub-matrix can be matched separately, and the results can be summarized at last.

[0057] In step 103, if the similarity exceeds a threshold, a first position of the similar video frame in the video to be retrieved and a second position of the similar video frame in the candidate video are determined based on the offset.

[0058] When the similarity exceeds the threshold, it can be confirmed that there are similar video frames between the two videos. At this time, the positions of the similar video frames in the two videos can be determined based on the offset. For example, the video to be retrieved and the candidate video are both 50 frames. When the similarity exceeds the threshold, if the absolute value of the offset is 5, it means that the first frame to the forty-fifth frame of one video is similar to the sixth frame to the fiftieth frame of the other video. As for whether the similar video frames correspond to the first frame to the forty-fifth frame or the sixth frame to the fiftieth frame in the video to be retrieved, it depends on the positive or negative of the offset, that is, the positive or negative of the offset indicates the direction of displacement.

[0059] In addition, when the offset is equal to zero, further judgment can be made based on the relationship between the similarity and the threshold. The similarity is positively correlated with the number of identical or similar elements between the feature matrices of the two videos. If the similarity is greater than the first threshold, it can be considered that the similarity between the video to be retrieved and the candidate video is very high, and at this time, it is determined that the video to be retrieved and the candidate video are the same; if the similarity is less than the first threshold and greater than the second threshold, wherein the first threshold is greater than the second threshold, then it is considered that the similar video frames existing in the video to be retrieved and the candidate video are in the middle position. In some examples, it can be achieved by segmenting the video to be retrieved and the candidate video, such as into N parts of the same size, N is greater than or equal to 2, and then performing step 102 on the feature matrix of the segmented video to determine the position of the similar video frames in the two videos according to the re-determined similarity and offset, wherein, if the matching result is still that the similarity is less than the first threshold and greater than the second threshold and the offset is equal to zero, then continue segmenting. It should be noted that the similarity in this embodiment can be a value in the interval [0, 1], then the first threshold can be a value close to 1, such as 0.8, 0.9, etc., and the second threshold can be a value less than the first threshold, such as 0.3, 0.4, etc. Of course, how to set these values ​​can also be determined according to the needs of the actual scenario, and this manual does not limit this.

[0060] From the previous steps, it can be seen that if the number of frames of the video to be retrieved and the candidate video are not equal, multiple sets of similarities and offsets can be obtained. Based on the multiple sets of offsets, all similar video frames in the two videos can be summarized. At this time, the matrix corresponding to the positions of all similar video frames can be segmented from the feature matrix of the video to be retrieved and the feature matrix of the candidate video, and then step 102 is executed for further verification to ensure the accuracy of video retrieval. For example, the video to be retrieved has a total of 50 frames, recorded as frames A1 to A50, and the candidate video has a total of 75 frames, recorded as frames B1 to B75. After executing step 102 to match the feature matrix corresponding to frames A1 to A50 with the feature matrix corresponding to frames B1 to B50, it is determined that frames A1 to A30 and frames B21 to B50 are repeated segments. After executing step 102 to match the feature matrix corresponding to frames A1 to A50 with the feature matrix corresponding to frames B26 to B75, it is determined that frames 6 to A40 and frames B26 to B60 are repeated segments. Summarize the two matching results, determine that frames A1 to A40 in the video frame to be retrieved and frames B21 to B60 in the candidate video are repeated segments, then the matrix corresponding to the position of frames A1 to A40 can be segmented from the feature matrix of the video to be retrieved, and the matrix corresponding to the position of frames B21 to B60 can be segmented from the feature matrix of the candidate video, and then step 102 is performed to match the two segmented matrices. If the re-determined similarity is greater than or equal to the first threshold mentioned above and the offset is equal to zero, it can be determined that the matching result is accurate. In addition, since the similarity is positively correlated with the number of identical or similar elements between the feature matrices of the two videos, the size of the similarity has a certain relationship with the number of frames of the similar video frames. Based on this, if the similarity is small, and the number of frames of the similar video frames determined according to the offset is large, the corresponding positions of the similar video frames can also be intercepted from the two feature matrices, and the phase correlation method is performed again to verify it.

[0061] In addition, when the similarity is less than a threshold, it can be considered that the similarity between the video to be retrieved and the candidate video is low, and the video to be retrieved can be stored in the video library, and the video fingerprint and feature matrix of the video to be retrieved can be stored in the database, thereby realizing automatic processing of the video to be retrieved and facilitating the operation of the video platform administrator.

[0062] The solution of this embodiment obtains the similarity between the two videos and the offset in time sequence of similar video frames in the two videos based on the cross power spectrum between the feature matrix of the video to be retrieved and the feature matrix of the candidate video obtained from the video library, so that when the similarity exceeds a threshold, the position of the similar video frame in the two videos can be determined based on the offset. This solution uses the phase correlation method to match the video to be retrieved with the video in the video library, without the need to make a judgment on each video frame, thereby improving the efficiency of video retrieval.

[0063] In order to explain the solution of this specification in more detail, a specific embodiment is introduced below:

[0064] In this embodiment, the video retrieval method of this specification is applied to the server of the video platform, and a video retrieval system is constructed in the server, such as Figure 2 As shown, Figure 2 is a schematic diagram of a video retrieval system shown in this embodiment, which mainly consists of a fingerprint generation module 21, a fingerprint retrieval module 22, a recheck module 23, a video library 24 and a video fingerprint library 25; wherein the fingerprint generation module 21 is used to generate a feature matrix and a video fingerprint for each video, and store the generated feature matrix and video fingerprint in the video fingerprint library 25; the fingerprint retrieval module 22 is used to find the most similar first K video fingerprints from the video fingerprint library 25 given the video fingerprint of the video to be retrieved, and determine the corresponding video as the candidate video; the recheck module 23 is used to match the video to be retrieved and the candidate video using the phase correlation method, so as to determine whether there are repeated segments and the location of the repeated segments; the video library 24 is used to store videos of the video platform; the video fingerprint library 25 is used to store the video fingerprints and feature matrices corresponding to the videos in the video library 24.

[0065] For the video A to be retrieved, the processing process of the system is as follows:

[0066] S201, generating a feature matrix and a video fingerprint of a video A to be retrieved by a fingerprint generation module;

[0067] Specifically, the frame rate of the video A to be retrieved is fixed to a preset frame rate. For the video A to be retrieved after the frame rate is fixed, image hash is selected as the frame-level feature of the video. The video A to be retrieved after the frame rate is fixed includes T1 video frames, and the frame-level features of each video frame have a total dimension of D. Therefore, the feature matrix of the video A to be retrieved is a T1×D matrix, denoted as matrix M1. The SVD decomposition is performed on the feature matrix to obtain the initial vector composed of the diagonal of its singular value matrix, and the target vector of the first N dimensions in the initial vector is intercepted as the video fingerprint of the video A to be retrieved;

[0068] S202, determining candidate video B through a fingerprint retrieval module;

[0069] Specifically, the video fingerprint is a vector of fixed length. The Faiss algorithm is used to compare the vector similarity of the video fingerprint of the video A to be retrieved with the video fingerprints of the videos in the video library, and sort them. The videos corresponding to the first K video fingerprints are determined as candidate videos B. The video fingerprint and feature matrix of the candidate video B are also generated by the fingerprint generation module and stored in the video fingerprint library in advance.

[0070] S203, determining the similarity and offset between the to-be-retrieved video A and the candidate video B through a recheck module;

[0071] Specifically, the phase correlation method is used to achieve fast alignment in the time dimension. The candidate video B is a set of multiple candidate videos. Taking one of the candidate videos B1 as an example, the candidate video B1 has a total of T2 video frames, so the feature matrix of the candidate video B1 is a T2×D matrix, recorded as matrix M2;

[0072] Step 1: Take the smaller of T1 and T2 as T0, and divide the feature matrix corresponding to the larger one. In this embodiment, since T1=50 and T2=75, the matrix M2 is divided into a sub-matrix M21 and a sub-matrix M22, and the sizes of the two sub-matrices are both T0×D; wherein the sub-matrix M21 is a matrix representing the features of the first frame to the fiftieth frame in the candidate video B1, and the sub-matrix M22 is a matrix representing the features of the twenty-sixth frame to the seventy-fifth frame in the candidate video B1;

[0073] The second step is to use the phaseCorrelate function to match the matrix M1 and the submatrix M21 to obtain the first set of values ​​returned by the phaseCorrelate function, which are the normalized maximum response value and the relative displacement, respectively. The normalized maximum response value represents the similarity between the two matrices, and the relative displacement represents the displacement required for the two matrices to reach the similarity. Similarly, the phaseCorrelate function is used to match the matrix M1 and the submatrix M22 to obtain the second set of values ​​returned, which are also the normalized maximum response value and the relative displacement.

[0074] Step 3: Use the set threshold to judge the maximum response value in the first group of values ​​and the second group of values. If the maximum response value exceeds the first threshold, it is considered that there is a repeated segment. The relative displacement at this time indicates the position of the repeated segment. The first group of values ​​and the second group of values ​​are summarized to obtain the position of the repeated segment in the matrix M1 and the matrix M2.

[0075] Step 4: Take the positions of repeated fragments from matrix M1 and matrix M2 as new matrices, and use phaseCorrelate function to match again. If the normalized maximum response value exceeds the second threshold, it is determined that the two sets of values ​​obtained in the second step are correct. Otherwise, divide matrix M1 and matrix M2 into N parts respectively and then execute the second step.

[0076] For other videos B2, B3, etc. in candidate video B, the same process from step 1 to step 4 is used to determine the similarity and offset;

[0077] S204, based on the determination result of the recheck module, processing the video A to be retrieved;

[0078] Specifically, if the maximum response values ​​between the video to be retrieved A and the candidate video B do not exceed the first threshold, the video to be retrieved A is stored in the video library, and the video fingerprint and feature matrix of the video to be retrieved A are stored in the video fingerprint library; otherwise, the video to be retrieved A is marked, and the marked content includes the identification of the candidate video B, the first position of the repeated segment in the video to be retrieved A, and the second position of the repeated segment in the candidate video B, and the marked video to be retrieved A is stored to wait for manual review by the administrator of the video platform.

[0079] Through the above process, the video retrieval system can quickly retrieve new videos uploaded by the publisher, which improves the operating efficiency and facilitates the administrator's processing. In addition, it should be noted that, through experiments, it is determined that the video retrieval system of this embodiment can be applied to a variety of business scenarios such as changes in video frame rate, changes in video resolution, adding watermarks or subtitles to the video screen, and adding not too long opening and ending credits to the video. The proportion of correct video detection is very high. Therefore, the video retrieval system of this embodiment has good robustness and versatility. In addition, the modules of the video retrieval system of this embodiment can be decoupled, so code or temporary data reuse can be achieved, thereby improving operating efficiency, and also facilitating the deployment of HTTP services, and facilitating large-scale deployment and calling.

[0080] Corresponding to the embodiments of the aforementioned method, this specification also provides embodiments of a video retrieval device and a terminal used therein.

[0081] The video retrieval device of this specification can be applied to computer devices, such as servers or terminal devices. The device embodiment can be implemented by software, or by hardware or a combination of software and hardware. Taking software implementation as an example, as a device in a logical sense, it is formed by the processor of the file processing in which it is located reading the corresponding computer program instructions in the non-volatile memory into the memory and running them. From the hardware level, if Figure 3 The figure is a hardware structure diagram of the computer device where the video retrieval device of the embodiment of this specification is located, except Figure 3 In addition to the processor 310, memory 330, network interface 320, and non-volatile memory 340 shown, the server or electronic device where the device 331 is located in the embodiment may also include other hardware according to the actual function of the computer device, which will not be described in detail.

[0082] Accordingly, an embodiment of the present specification further provides a computer storage medium, in which a program is stored. When the program is executed by a processor, the method in any of the above embodiments is implemented.

[0083] The embodiments of this specification may take the form of a computer program product implemented on one or more storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing program code. Computer-usable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include but are not limited to: phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0084] like Figure 4 As shown, Figure 4 is a block diagram of a video retrieval device according to an exemplary embodiment of the present specification, the device comprising:

[0085] An acquisition module 41 is used to acquire a feature matrix of a video to be retrieved;

[0086] A matching module 42 is used to obtain the similarity and offset between the video to be retrieved and the candidate video based on the cross power spectrum between the feature matrix of the video to be retrieved and the feature matrix of the candidate video; the candidate video is obtained from the video library; the offset represents the offset in time sequence between similar video frames in the video to be retrieved and the candidate video;

[0087] The determination module 43 is configured to determine, if the similarity exceeds a threshold, a first position of the similar video frame in the video to be retrieved and a second position of the similar video frame in the candidate video based on the offset.

[0088] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0089] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0090] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0091] Those skilled in the art will readily appreciate other embodiments of the specification after considering the specification and practicing the invention claimed herein. The specification is intended to cover any variations, uses or adaptations of the specification that follow the general principles of the specification and include common knowledge or customary techniques in the art that are not claimed in the specification. The specification and examples are to be considered exemplary only, and the true scope and spirit of the specification are indicated by the following claims.

[0092] It should be understood that the present description is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present description is limited only by the appended claims.

[0093] The above description is only a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of this specification should be included in the scope of protection of this specification.

Claims

1. A video retrieval method, characterized in that: The method comprises: Obtain the feature matrix of the video to be retrieved; Based on the cross power spectrum between the feature matrix of the video to be retrieved and the feature matrix of the candidate video, the similarity and offset between the video to be retrieved and the candidate video are obtained; the candidate video is obtained from a video library; the offset represents the temporal offset of similar video frames in the video to be retrieved and the candidate video; If the similarity exceeds a threshold, determining a first position of the similar video frame in the video to be retrieved and a second position of the similar video frame in the candidate video based on the offset, including: If the similarity is greater than or equal to a first threshold, and the offset is equal to zero, determining that the similar video frames include all video frames of the video to be retrieved and the candidate video; If the similarity is less than the first threshold and greater than the second threshold, and the offset is equal to zero, the video to be retrieved and the candidate video are segmented, and the similarity and offset between the segmented videos are redetermined to determine the position of the similar video frame according to the redetermined similarity and offset; wherein the first threshold is greater than the second threshold.

2. The method according to claim 1, characterized in that The candidate video is obtained from a video library based on the video fingerprint of the video to be retrieved.

3. The method according to claim 2, characterized in that The video fingerprint of the video to be retrieved / candidate video is composed based on singular values ​​obtained by performing singular value decomposition on the feature matrix of the video to be retrieved / candidate video.

4. The method according to claim 3, characterized in that The video fingerprint of the video to be retrieved / candidate video is obtained based on the following method: Performing singular value decomposition on the feature matrix of the video to be retrieved / candidate video, and forming an initial vector with the obtained singular values; A preset dimension in the initial vector is intercepted as a target vector, and the target vector is determined as a video fingerprint of the video to be retrieved / candidate video.

5. The method according to claim 1, characterized in that Based on the cross power spectrum between the feature matrix of the video to be retrieved and the feature matrix of the candidate video, the similarity and offset between the video to be retrieved and the candidate video are obtained, including: Performing inverse Fourier transform on the cross power spectrum between the feature matrix of the video to be retrieved and the feature matrix of the candidate video to obtain an impulse function; A peak point is obtained based on the impulse function, and the similarity and offset between the video to be retrieved and the candidate video are obtained according to the coordinates of the peak point.

6. The method according to claim 1, characterized in that Before obtaining the similarity and offset between the video to be retrieved and the candidate video based on the cross power spectrum between the feature matrix of the video to be retrieved and the feature matrix of the candidate video, the method includes: If the number of frames of the video to be retrieved is not equal to the number of frames of the candidate video, at least one of the feature matrix of the candidate video and the feature matrix of the video to be retrieved is segmented so that the feature matrix of the candidate video and the feature matrix of the video to be retrieved after segmentation have the same number of rows and columns.

7. The method according to claim 1, characterized in that The method further comprises: Determine a first position of the similar video frame in the video to be retrieved and a second position of the similar video frame in the candidate video as an initial matching result; Segmenting a first matrix corresponding to the first position from the feature matrix of the video to be retrieved, and segmenting a second matrix corresponding to the second position from the feature matrix of the candidate video; Based on the cross power spectrum between the first matrix and the second matrix, obtaining the target similarity and target offset between the first matrix and the second matrix; If the target similarity is greater than or equal to the first threshold, and the target offset is equal to zero, it is determined that the initial matching result is correct.

8. A computer device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

9. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.