Retrieval method and device of repeated video, electronic equipment and medium

Abstract

One or more embodiments of the invention provide a repeated video retrieval method and apparatus, an electronic device and a medium. The method comprises the steps of extracting key frames included in a target video and image features of each key frame; for each key frame of the target video, performing the following operations: performing feature similarity comparison on image features of the key frame and image features stored in a database, and retrieving to obtain frame information of a similar key frame similar to the key frame of the target video, the frame information comprising a video identifier of a candidate video to which the similar key frame belongs; based on the video identifier, aggregating each similar key frame belonging to the same candidate video to obtain a candidate video set, the candidate video set comprising at least one candidate video, and each candidate video comprising at least one similar key frame; and performing video matching comparison on any candidate video and the target video, and determining a repeated video corresponding to the target video from the candidate video set.

Description

Technical Field

[0001] One or more embodiments of the present specification relate to the field of network technology, and in particular, to a method, device, electronic device, and medium for retrieving duplicate videos. Background Art

[0002] Duplicate video retrieval can be involved in many scenarios. For example, in content security review, a base library consisting of illegal video content can be built. If a newly released video can be found as a duplicate in the base library, it means that the newly released video is illegal content, and the newly released video can be blocked to avoid exposure. Other application scenarios are not listed in detail.

[0003] In the related art, for the retrieval of duplicate videos, a feature is extracted from the entire video, and then the database is searched based on the feature. However, this method can usually only be used to retrieve 100% duplicate videos. If the video has been edited, cropped, etc., it cannot be retrieved. For example, a one-minute video with 30 seconds of content cut out of it cannot be retrieved from each other. In real scenes, many relocations and plagiarisms will involve confrontational behaviors. The plagiarists will edit and modify the videos to disguise the plagiarism. Therefore, it is necessary to improve the duplicate video retrieval method. Summary of the invention

[0004] In view of this, one or more embodiments of the present specification provide a method, apparatus, electronic device, and medium for retrieving duplicate videos to improve the retrieval recall rate of duplicate videos and improve the retrieval efficiency of duplicate videos.

[0005] To achieve the above objectives, one or more embodiments of this specification provide the following technical solutions:

[0006] According to a first aspect of an embodiment of the present specification, a method for retrieving duplicate videos is provided, the method being used to retrieve whether there is a duplicate video that is duplicate of a target video in a database; the method comprising:

[0007] Extracting key frames included in the target video and image features of each key frame;

[0008] For each key frame of the target video, the following operations are performed: by comparing the image features of the key frame with the image features stored in the database, frame information of a similar key frame similar to the key frame of the target video is retrieved, wherein the frame information includes: a video identifier of a candidate video to which the similar key frame belongs;

[0009] Based on the video identifier, similar key frames belonging to the same candidate video are aggregated to obtain a candidate video set, wherein the candidate video set includes at least one candidate video, and each candidate video includes at least one similar key frame;

[0010] Perform video matching comparison between any candidate video and the target video, and determine a duplicate video corresponding to the target video from the candidate video set.

[0011] According to a second aspect of the embodiments of this specification, a device for retrieving duplicate videos is provided, the device being used to retrieve whether there is a duplicate video that is duplicated with a target video in a database; the device comprising:

[0012] An information acquisition module, used to extract key frames included in the target video and image features of each key frame;

[0013] The similar frame acquisition module is used to perform the following operations on each key frame of the target video: by comparing the image features of the key frame with the image features stored in the database, retrieve the frame information of the similar key frame similar to the key frame of the target video, and the frame information includes: the video identifier of the candidate video to which the similar key frame belongs;

[0014] A video acquisition module, configured to aggregate similar key frames belonging to the same candidate video based on the video identifier to obtain a candidate video set, wherein the candidate video set includes at least one candidate video, and each candidate video includes at least one similar key frame;

[0015] The video matching module is used to compare any candidate video with the target video for video matching, and determine the duplicate video corresponding to the target video from the candidate video set.

[0016] According to a third aspect of the embodiments of this specification, an electronic device is provided, including:

[0017] processor;

[0018] a memory for storing processor-executable instructions;

[0019] The processor implements the method described in any embodiment of this specification by running the executable instructions.

[0020] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, on which computer instructions are stored, and when the instructions are executed by a processor, the method described in any embodiment of this specification is implemented.

[0021] According to a fifth aspect of the embodiments of this specification, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method described in any embodiment of this specification is implemented.

[0022] The duplicate video retrieval method, device, electronic device and medium of the embodiments of this specification first perform feature similarity comparison on key frames and then aggregate similar key frames to obtain a candidate video set, which can greatly reduce the number of videos that need to be matched. For example, if a database stores a large number of videos, it is not necessary to perform video matching comparison on each of these videos with the target video. Instead, it is only necessary to match and compare the aggregated and screened candidate videos with the target video, thereby narrowing the matching range and improving the retrieval efficiency of duplicate videos. In particular, for some scenarios with high real-time requirements, duplicate videos can be retrieved quickly in massive data. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in one or more embodiments of the present disclosure or related technologies, the drawings required for use in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in one or more embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0024] Figure 1 The present invention is a flowchart of a video storage provided by an exemplary embodiment.

[0025] Figure 2 The present invention is a schematic diagram of a video storage process provided by an exemplary embodiment.

[0026] Figure 3 It is a flowchart of a method for retrieving duplicate videos provided by an exemplary embodiment.

[0027] Figure 4 It is a process diagram of a method for retrieving duplicate videos provided by an exemplary embodiment.

[0028] Figure 5 It is a flowchart of a video matching provided by an exemplary embodiment.

[0029] Figure 6 It is a schematic structural diagram of a repeated video retrieval device provided by an exemplary embodiment.

[0030] Figure 7 is a schematic diagram of an electronic device provided by an exemplary embodiment. DETAILED DESCRIPTION

[0031] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with one or more embodiments of this specification. Instead, they are merely examples of devices and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0032] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may be combined into a single step for description in other embodiments.

[0033] The embodiments of the present specification provide a method for retrieving duplicate videos, which can be applied to retrieving duplicate videos.

[0034] The following are two examples of application scenarios for duplicate video retrieval:

[0035] In an example scenario, a base library consisting of illegal video content can be built during content security review. If a newly released video can be found as a duplicate in the base library, it means that the newly released video is illegal content, and the newly released video can be blocked to avoid exposure.

[0036] In another example scenario, in ecological content governance, a base library is built for all historical videos that have been released. If duplicate content can be retrieved from the base library for a newly released video, the newly released video may involve plagiarism or copying, and the newly released video can be intercepted and reviewed.

[0037] As described above, the retrieval of duplicate videos usually requires the construction of a base library, and when searching for duplicate videos, the retrieval can be performed from the base library. Similarly, the retrieval method for duplicate videos in the embodiment of this specification also has a constructed base library, and the base library is searched to determine whether there are duplicate videos. In the following description, the construction of the base library used in the retrieval method will be described first, and then the process of retrieving duplicate videos based on the base library will be described.

[0038]

Construction of base library

[0039] See also Figure 1 , Figure 1This is a schematic diagram of a video storage process provided by an exemplary embodiment. This embodiment describes the process of storing a video in a base library. The following processing may be included:

[0040] In step 100, a first video to be stored in a database is obtained.

[0041] In this embodiment, the database is a base database for repeated video retrieval. For example, the database may be an ANN (Approximate Nearest Neighbor) database, which can support rapid search for vector features similar to the desired features from a large number of features.

[0042] The video to be stored in the database can be called the first video. For example, in the scenario of ecological content governance, the first video can be a historical video that has been released. For another example, in the scenario of content security review, the first video can be an illegal video.

[0043] In step 102, key frames of the first video and image features of each key frame are extracted.

[0044] The key frame may be an important picture representing the video content extracted from the first video. In this step, the key frame may be extracted from the first video by a key frame extraction algorithm, and an image feature may be extracted for each key frame. For example, the image feature may be an image feature of a two-dimensional vector.

[0045] In an example implementation, the process of extracting key frames by a key frame extraction algorithm may include: calculating the inter-frame differences between consecutive frames in the first video. For example, the inter-frame differences may be calculated by using methods such as histogram comparison and pixel difference. Then, according to the size of the above-mentioned inter-frame differences, key frames are selected based on certain criteria. For example, a difference threshold may be set. When the difference between two consecutive frames exceeds this threshold, it is considered that a scene change has occurred, and the latter of the two frames is marked as a key frame. In addition, deduplication screening and other processing may be performed to ultimately obtain multiple key frames included in the first video.

[0046] In an exemplary implementation, an image feature is extracted from each key frame, and a convolutional neural network may be used to extract an image feature of a two-dimensional vector from each key frame. This embodiment does not limit the method of extracting image features.

[0047] In step 104, the image feature and frame information of the key frame corresponding to the image feature are stored in the database, and the frame information includes: a video identifier of the first video to which the key frame belongs.

[0048] This step can be stored. Please refer to Figure 2 As shown, Figure 2 This is a schematic diagram of a video storage process provided by an exemplary embodiment. A plurality of key frames can be extracted from the first video, for example, key frame 1, key frame 2, key frame 3, etc., and corresponding image features can be extracted for each key frame through the aforementioned steps. Finally, the image features and the frame information of the key frames corresponding to the image features can be stored in a database.

[0049] The frame information of the key frame corresponding to the image feature may include but is not limited to the following information:

[0050] 1) Frame identification of key frame;

[0051] The frame identifier is used as a unique ID of a key frame in the first video.

[0052] This embodiment does not limit the definition method of the frame identifier. For example, before extracting the key frame of the first video, each video frame in the first video can be added with a serial number according to the time sequence of the video. After extracting the video frame as the key frame, the serial number can be used as the frame identifier of the key frame. Figure 2 The "1", "2", and "3" in the "key frame 1, key frame 2, key frame 3" shown in the figure can be called frame identifiers.

[0053] 2) the video identifier of the first video to which the key frame belongs;

[0054] The video identifier is used as the unique ID of the first video.

[0055] This embodiment does not limit the definition method of the video identifier. For example, "A", "B", and "C" in video A, video B, and video C can be called video identifiers.

[0056] 3) The total number of key frames extracted from the first video.

[0057] For example, if a total of 8 key frames are extracted from the first video, then the total number of key frames is 8.

[0058] Based on the above, the storage information of the first video in the database is exemplified as follows through Table 1:

[0059] Table 1 Video storage information in the database

[0060]

[0061]

[0062] It should be noted that this embodiment does not limit the specific content of the frame information stored in the database, wherein the video identifier of the first video to which the key frame corresponding to the image feature belongs may be stored. In addition, the frame identifier of the key frame and some data to be used in the subsequent retrieval process of repeated videos, such as the total number of key frames, etc., may also be stored.

[0063]

Retrieval of duplicate videos

[0064] After the database is constructed according to the above-mentioned base database construction method, the duplicate video retrieval method of this embodiment can be used to retrieve duplicate videos. Figure 3 , Figure 3 The following is a flow chart of a method for retrieving duplicate videos provided by an exemplary embodiment. In this embodiment, it is possible to search whether there is a video in the database that is duplicate of the target video.

[0065] The retrieval method may include the following processing:

[0066] In step 300, key frames included in the target video and image features of each key frame are extracted.

[0067] In this embodiment, the video to be checked for duplication can be referred to as a target video. For example, if a user Xiao Wang wants to publish a new video on a certain video platform, the search method of this embodiment can be used to search the database to see if there is a video that is duplicated with the video that Xiao Wang has just published. Then the video that Xiao Wang has just published can be referred to as a target video.

[0068] In this step, the key frames included in the target video and the image features of each key frame can be extracted. The key frame extraction method and the image feature extraction method can refer to the above-mentioned embodiment.

[0069] Please refer to Figure 4 As shown, the target video to be checked for duplicate content in this embodiment may be: video Q, where Q may be referred to as the video identifier of the target video. Furthermore, it is assumed that three key frames are extracted from video Q, namely key frame Q1, key frame Q2, and key frame Q3, and the image features of each key frame are extracted by a convolutional neural network.

[0070] Table 2 below illustrates the image features corresponding to each key frame:

[0071] Table 2 Image features corresponding to key frames

[0072] Keyframes Image features Keyframe Q1 Image feature T1 Keyframe Q2 Image feature T2 Keyframe Q3 Image feature T3

[0073] In step 302, for each key frame of the target video, the following operation is performed: by performing feature similarity comparison between the image features of the key frame and the image features stored in the database, frame information of similar key frames similar to the key frames of the target video is retrieved, and the frame information includes: the video identifier of the candidate video to which the similar key frame belongs.

[0074] In this step, for each key frame of the target video, similar frames of the key frame can be retrieved by comparing the feature similarity of the image features.

[0075] In an example implementation, a database stores many image features, and the image features of the key frames of the target video can be compared with the image features included in the database for feature similarity, and the similarity values ​​between the image features can be calculated. Since the image features are extracted based on the key frames of the video, the similarity values ​​between the image features actually also represent the similarity between the key frames.

[0076] On the basis of obtaining the similarity values ​​between the above-mentioned image features, at least some image features can be selected from multiple image features whose similarity values ​​reach a preset similarity threshold and determined as similar image features corresponding to the image features of the key frames of the target video.

[0077] For example, suppose that after feature similarity comparison between image features, the following similarity values ​​"0.8, 0.89, 0.92, 0.95, 0.76, 0.83" are obtained. If the similarity threshold is set to 0.85, then "0.89, 0.92, 0.95" all meet the "similarity value reaches the preset similarity threshold". However, whether all image features that reach the similarity threshold are screened as similar image features that are similar to the image features of the key frames of video Q can be determined according to the retrieval settings. That is, users can set the screening conditions for image feature similarity according to their own retrieval needs, and this embodiment does not limit this.

[0078] For example, if the user wants to improve the recall rate of the retrieval and is not too strict on the similarity requirement, all image features that reach the similarity threshold can be determined as similar image features similar to the image features of the key frames of video Q, or the similarity threshold can be set slightly lower.

[0079] For another example, if the user has a high requirement for the similarity of the retrieved videos, the similarity threshold can be set higher, and all image features that reach the similarity threshold can be sorted, and image features corresponding to the first M similarity values ​​after sorting can be selected as similar image features, where M is a natural number.

[0080] As described above, based on the similarity values ​​between image features, at least a portion of features that meet the screening conditions for image feature similarity can be selected as similar image features corresponding to the image features of the key frames of the target video, and the key frames corresponding to the similar image features can be called similar key frames.

[0081] Please refer to Figure 4 In the example, based on the image feature T1, the database is searched, and by calculating the similarity values ​​between the image features, the similar image features of the image feature T1 are retrieved, including: image feature A1, image feature B3, image feature B4, image feature C5. Among them, the database may return the frame information of the similar key frames corresponding to these similar image features, as shown in Table 3 below.

[0082] Table 3 Frame information for retrieving similar key frames

[0083]

[0084] As shown in the above table, for example, the similar key frame corresponding to the image feature A1 is "Key frame 1 of video A", where "Key frame 1 of video A" is called a similar key frame, video A is called a candidate video to which the similar key frame belongs, and A is the video identifier of the candidate video. The same is true for the others, which will not be described in detail. That is, there are multiple similar key frames corresponding to the key frame Q1 of the target video Q, including: "Key frame 1 of video A", "Key frame 3 of video B", "Key frame 4 of video B", and "Key frame 5 of video C".

[0085] In the same way, similar key frames of key frame Q2 and similar key frames of key frame Q3 can be retrieved from the database, respectively, which can be combined with reference to the above-mentioned Table 3.

[0086] In this step, when searching the database and returning the frame information of the similar key frame corresponding to each key frame of the target video, the frame information may include the above: the video identifier of the candidate video to which the similar key frame belongs, and the frame identifier of the similar key frame. For example, in "Video A—Key Frame 1", the frame identifier of the similar key frame is "1", the candidate video to which it belongs is "Video A", and the video identifier of the candidate video is "A". That is, through the returned frame information, it can be known which frame of which other video is similar to the key frame of the target video.

[0087] In addition, in order to facilitate the matching and comparison of videos in subsequent steps, the frame information of similar key frames returned in this step may also include: the total number of key frames and the similarity score.

[0088] Still taking "Video A—Key Frame 1" as an example, in addition to returning the video ID of the candidate video mentioned above and the frame ID of the similar key frame, it can also return:

[0089] 1) The total number of key frames included in video A, for example, the total number of key frames is 6.

[0090] 2) The similarity value between the key frame of the target video and the similar key frame.

[0091] The similarity value is obtained by comparing the image features of the target video's key frame with the image features of the similar key frame. For example, still taking "Video A—Key Frame 1" as an example, the key frame 1 of Video A is a similar key frame of the key frame Q1 of the target video, and the similarity value between the key frame 1 and the key frame Q can be 0.98.

[0092] Similarly, a similarity value can be output between each similar key frame and its corresponding key frame of the target video. Please refer to the example of Table 3 above.

[0093] In this embodiment, the search conditions of similar key frames can be set, for example, a similarity threshold can be set, and the number of similar key frames returned by the database can also be set. In actual implementation, users can adjust the above search conditions according to their own search needs, which is flexible and convenient.

[0094] In step 304, based on the video identifier, similar key frames belonging to the same candidate video are aggregated to obtain a candidate video set, wherein the candidate video set includes at least one candidate video, and each candidate video includes at least one similar key frame.

[0095] In this step, similar key frames belonging to the same candidate video may be aggregated based on the video identifier of the candidate video to obtain a candidate video set, which may include at least one candidate video. Each candidate video includes multiple similar key frames, and also includes key frames of the target video corresponding to each similar key frame, and similarity values ​​between frames and other information.

[0096] For example, taking the video C in Table 3 above as an example, based on the video identifier of video C, the similar key frames belonging to video C are aggregated to obtain: video C-->{key frame 5 of video C: key frame Q1 of video Q: similarity value 0.95; key frame 1 of video C: key frame Q2 of video Q: similarity value 0.98; key frame 3 of video C: key frame Q3 of video Q: similarity value 0.96}.

[0097] Please refer to Table 3 above. After aggregation, the following candidate videos can be obtained:

[0098] Table 4 Candidate video set

[0099]

[0100] In step 306, any candidate video is compared with the target video for video matching, and a duplicate video corresponding to the target video is determined from the candidate video set.

[0101] In this step, each candidate video in the candidate video set may be compared with the target video for video matching, so as to determine the matched candidate video as a duplicate video of the target video.

[0102] The following uses the KM (Kuhn-Munkres) algorithm (also known as the Hungarian algorithm) to determine the matching relationship between two videos as an example, but it is understandable that the actual implementation is not limited to this. In the following description, the KM algorithm is briefly described and no further details are given.

[0103] Specifically, you can combine Figure 5 As shown, Figure 5 This is a schematic diagram of a video matching process provided by an exemplary embodiment. It may include the following processing:

[0104] In step 500, a bipartite graph is constructed according to the key frame set of the target video and the key frame set of the candidate video.

[0105] This step can construct a bipartite graph, wherein the bipartite graph may include two sets, one of which is a key frame set of the target video, including all key frames in the target video. The other set is a key frame set of the candidate video, including all key frames in the candidate video. It should be noted that all key frames in the candidate video include the aforementioned similar key frames and non-similar key frames.

[0106] For example, the candidate video - "key frame 1", "key frame 3" and "key frame 4" in video B are similar key frames to the key frames of the target video, but video B also includes "key frame 2" and "key frame 5". Then when constructing the bipartite graph, "key frame 1", "key frame 2", "key frame 3", "key frame 4" and "key frame 5" in video B are all put into one of the sets of the bipartite graph.

[0107] In addition, in the bipartite graph, there is a connecting edge between the key frame of the target video and the key frame of the candidate video, and the weight of the connecting edge represents the similarity between the frames, and the weight is determined based on the similarity value.

[0108] For example, if the similarity value between key frame 3 of video B and key frame Q1 of video Q is 0.97, then in the bipartite graph, the weight value of the edge connecting the two key frames can be 0.97. Of course, it can be understood that the weight of the edge represents the similarity between the frames, and the weight can also be other values ​​determined based on the similarity value, for example, the similarity value can be converted into the Euclidean distance, and the Euclidean distance can be used as the weight.

[0109] For key frames that do not have similarity values ​​in the feature similarity comparison in the above step, their weight values ​​can be set to indicate that the similarity between the frames is very low. For example, in the example of using similarity values ​​as weights, key frame 3 of video B and key frame Q3 of video Q are not similar, and their weights can be set to a lower value.

[0110] In step 502, matching frame pairs between the key frame set of the target video and the key frame set of the candidate video are calculated based on the bipartite graph.

[0111] In this step, based on the construction of the bipartite graph, the KM algorithm can be applied to find the maximum weight matching and find the matching frame pairs, that is, the matching results output by the KM algorithm are used to indicate which frames in the two sets are most matched with which frames. When searching, the KM algorithm assigns a top label to each part of the bipartite graph and finds an augmenting path to achieve the search for matching frame pairs.

[0112] Each matching frame pair includes: a key frame of the target video and a similar key frame of the candidate video.

[0113] For example, according to Table 4, the key frame Q1 of the target video Q is similar to the key frame 3 of the video B, and the key frame Q1 is also similar to the key frame 4 of the video B. It is possible that after the matching comparison of the KM algorithm, in the output result of the KM algorithm, the key frame Q1 is similar to the key frame 3 of the video B, but the key frame Q1 is not a matching frame pair with the key frame 4 of the video B, that is, in the process of feature similarity comparison, the similarity between the key frame Q1 and the key frame 4 of the video B is an erroneous comparison result. This is equivalent to further screening the results of the feature similarity comparison in the previous step through the KM algorithm to screen out more accurate matching frame pairs.

[0114] For example, for the candidate videos in Table 4, after being processed by the KM algorithm, the matching frame pairs output by KM may be the ones that have the following two similar key frames removed, "key frame 3 of video A --- key frame Q3", and "key frame 4 of video B --- key frame Q1", compared to Table 4.

[0115] In step 504, if it is determined that the candidate video matches the target video according to the matching frame pair, then the candidate video and the target video are duplicate videos.

[0116] Based on the matching frame pairs obtained by the KM algorithm in the previous step, this step can determine whether the candidate video matches the target video based on these matching frame pairs. Specifically, each candidate video can be matched and compared with the target video to determine whether the two match, wherein the matching comparison is based on the matching frame pairs between the two videos.

[0117] Exemplarily, matching information may be obtained based on matching frame pairs. Some matching information is listed below, but is not limited thereto in actual implementation:

[0118] 1) The number of matching frame pairs.

[0119] For example, taking video A as an example, KM outputs three matching frame pairs.

[0120] 2) The ratio of the key frames in the matching frame pair to the total number of key frames in the target video.

[0121] For example, in the case of three matching frame pairs in video A, that is, there are 3 key frames in video A that belong to the matching frame pairs, and there are also 3 key frames in the target video Q that belong to the matching frame pairs. The total number of key frames in the target video Q is 6, so the ratio of key frames in the matching frame pairs to the total number of key frames in the target video is 50%.

[0122] 3) The ratio of similar key frames in the matching frame pair to the total number of key frames in the candidate video to which it belongs.

[0123] For example, there are 3 key frames in video A that belong to the frames in the matching frame pair, and these frames can be called similar key frames. The total number of key frames in video A is 5, and the ratio of similar key frames to the total number of key frames in the candidate video to which they belong is 3 / 5=60%.

[0124] 4) Similarity information determined based on similarity values ​​of the matching frame pairs.

[0125] The similarity information may be determined based on a similarity value.

[0126] In an exemplary implementation, in the above example, there are 3 matching frame pairs between video A and target video Q. If the similarity values ​​of these three matching frame pairs are "0.98, 0.96, 0.99" respectively, then the similarity value can be the average of these three similarity values, or it can also be the sum of the three similarity values, etc.

[0127] If the matching information between the candidate video and the target video meets the matching condition, it is determined that the candidate video matches the target video. This embodiment does not limit the setting of the matching condition, and the user can determine it according to actual business needs.

[0128] Exemplarily, it can be set as "if the number of matching frame pairs reaches 3, and the ratio of key frames in the matching frame pairs to the total number of key frames of the target video and the candidate video reaches 60%, and the value of the similarity information is higher than 0.97, then it is determined that the two videos are matched."

[0129] The method for retrieving duplicate videos of this embodiment first performs feature similarity comparison based on the image features of key frames, then aggregates similar key frames belonging to the same video to obtain a candidate video set, and then performs video matching comparison between each candidate video in the candidate video set and the target video to finally obtain duplicate videos. This method has the following advantages:

[0130] 1) To retrieve duplicate videos through the key frames of the video, only the key frames need to be similar. Compared with the method of extracting a feature from the entire video to search the database, even if the video is slightly edited and cropped, it can still be retrieved through this method, thereby better preventing plagiarism and intercepting illegal videos.

[0131] 2) By first performing feature similarity comparison on key frames and then aggregating similar key frames to obtain a set of candidate videos, the number of videos that need to be matched can be greatly reduced. For example, if a large number of videos are stored in the database, there is no need to compare each of these videos with the target video for video matching. Instead, only the aggregated and screened candidate videos need to be matched and compared with the target video, thereby narrowing the matching range and improving the retrieval efficiency of duplicate videos. Especially for some scenarios with high real-time requirements, duplicate videos can be retrieved quickly in massive data.

[0132] 3) First, similar key frames are obtained by feature similarity comparison, and then candidate videos are obtained by aggregating similar key frames belonging to the same candidate video, so that the recall rate of similar key frames can be improved by similarity comparison of image features of key frames, thereby improving the recall rate of candidate videos; at the same time, by performing video matching comparison between any candidate video and the target video, the accuracy of duplicate video retrieval is guaranteed.

[0133] In order to implement the duplicate video retrieval method of the embodiment of this specification, this embodiment also provides a duplicate video retrieval device. Figure 6 FIG. 1 is a schematic diagram of a structure of a duplicate video retrieval device provided by an exemplary embodiment, and the device is used to retrieve whether there is a duplicate video that is the same as the target video in the database. Figure 6As shown, the device may include: an information acquisition module 61, a similar frame acquisition module 62, a video acquisition module 63 and a video matching module 64.

[0134] The information acquisition module 61 is used to extract key frames included in the target video and image features of each key frame.

[0135] The similar frame acquisition module 62 is used to perform the following operation on each key frame of the target video: by comparing the image features of the key frame with the image features stored in the database, the frame information of the similar key frame similar to the key frame of the target video is retrieved, and the frame information includes: the video identifier of the candidate video to which the similar key frame belongs.

[0136] The video acquisition module 63 is used to aggregate similar key frames belonging to the same candidate video based on the video identifier to obtain a candidate video set, wherein the candidate video set includes at least one candidate video, and each candidate video includes at least one similar key frame.

[0137] The video matching module 64 is used to perform video matching comparison between any candidate video and the target video, and determine a duplicate video corresponding to the target video from the candidate video set.

[0138] In one example, the similar frame acquisition module 62, when used to retrieve frame information of similar key frames similar to the key frames of the target video by performing feature similarity comparison on the image features of the key frames with image features stored in a database, includes: performing feature similarity comparison on the image features of the key frames of the target video with multiple image features included in the database to obtain similarity values ​​between the image features; selecting at least part of the image features from the multiple image features whose similarity values ​​reach a preset similarity threshold, and determining them as similar image features corresponding to the image features of the key frames of the target video; and acquiring frame information of similar key frames corresponding to the similar image features from the database.

[0139] In one example, the frame information further includes: a similarity value between the key frame of the target video and the similar key frame, wherein the similarity value is obtained by comparing the image features of the key frame of the target video with the image features of the similar key frame. The video matching module 64, when used to compare any candidate video with the target video for video matching and determine the duplicate video corresponding to the target video from the candidate video set, includes: constructing a bipartite graph based on the key frame set of the target video and the key frame set of the candidate video; in the bipartite graph, there is a connecting edge between the key frame of the target video and the key frame of the candidate video, and the weight of the connecting edge represents the similarity between the frames, and the weight is determined based on the similarity value; based on the bipartite graph, calculating the matching frame pairs between the key frame set of the target video and the key frame set of the candidate video, each matching frame pair includes: a key frame of the target video and a similar key frame of the candidate video; if the candidate video is determined to match the target video based on the matching frame pair, then the candidate video and the target video are duplicate videos.

[0140] In one example, the frame information also includes: the total number of key frames in the candidate video. The video matching module 64, when used to determine that the candidate video matches the target video based on the matching frame pairs, includes: obtaining at least a portion of the following matching information based on the matching frame pairs: the number of matching frame pairs, the ratio of the key frames in the matching frame pairs to the total number of key frames in the target video to which they belong, the ratio of similar key frames in the matching frame pairs to the total number of key frames in the candidate video to which they belong, and similarity information determined based on the similarity values ​​of the matching frame pairs; if the matching information between the candidate video and the target video meets the matching conditions, then determining that the candidate video matches the target video.

[0141] In one example, the information acquisition module 61 is also used to: acquire a first video to be stored in the database; extract key frames of the first video and image features of each of the key frames; store the image features and frame information of the key frames corresponding to the image features in the database, wherein the frame information includes: a video identifier of the first video to which the key frames belong.

[0142] For example, the frame information of the key frame corresponding to the image feature also includes the total number of key frames extracted from the first video.

[0143] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0144] For the device embodiment, since it basically corresponds to the method embodiment, the relevant parts can refer to the partial description of the method embodiment. The device embodiment described above is only schematic, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this specification. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0145] like Figure 7 As shown, Figure 7 A hardware structure diagram of an electronic device in which the navigation broadcast device of an embodiment of the specification is located is shown, and the device may include: a processor 710, a memory 720, an input / output interface 730, a communication interface 740, and a bus 750. The processor 710, the memory 720, the input / output interface 730, and the communication interface 740 are connected to each other in communication within the device through the bus 750.

[0146] The processor 710 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification. The processor implements the above method by running executable instructions.

[0147] The memory 720 for storing processor executable instructions can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 720 can store operating systems and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 720.

[0148] The input / output interface 730 is used to connect the input / output module to realize information input and output. The input / output module can be configured as a component in the device (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0149] The communication interface 740 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0150] The bus 750 comprises a pathway that transmits information between the various components of the device (eg, the processor 710 , the memory 720 , the input / output interface 730 , and the communication interface 740 ).

[0151] It should be noted that, although the above device only shows the processor 710, the memory 720, the input / output interface 730, the communication interface 740 and the bus 750, in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include the components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0152] The embodiments of this specification also provide a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method of any of the above embodiments is implemented.

[0153] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0154] The embodiments of the present specification also provide a computer program, which is used to implement the method described in any embodiment of the present specification when executed.

[0155] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0156] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0157] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0158] The above description is merely a preferred embodiment of one or more embodiments of the present specification and is not intended to limit one or more embodiments of the present specification. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of one or more embodiments of the present specification shall be included in the scope of protection of one or more embodiments of the present specification.

Claims

1. A method for retrieving duplicate videos, characterized in that: The method is used to retrieve whether there is a duplicate video in the database that is duplicate of the target video; the method comprises: Extracting key frames included in the target video and image features of each key frame; For each key frame of the target video, the following operations are performed: by comparing the image features of the key frame with the image features stored in the database, frame information of a similar key frame similar to the key frame of the target video is retrieved, wherein the frame information includes: a video identifier of a candidate video to which the similar key frame belongs; Based on the video identifier, similar key frames belonging to the same candidate video are aggregated to obtain a candidate video set, wherein the candidate video set includes at least one candidate video, and each candidate video includes at least one similar key frame; Perform video matching comparison between any candidate video and the target video, and determine a duplicate video corresponding to the target video from the candidate video set.

2. The method according to claim 1, characterized in that The retrieving frame information of a similar key frame similar to the key frame of the target video by performing feature similarity comparison between the image features of the key frame and the image features stored in the database comprises: Performing feature similarity comparisons on the image features of the key frame of the target video and the plurality of image features included in the database to obtain similarity values ​​between the image features; Selecting at least some of the image features from the plurality of image features whose similarity values ​​reach a preset similarity threshold value, and determining them as similar image features corresponding to the image features of the key frames of the target video; From the database, frame information of similar key frames corresponding to the similar image features is obtained.

3. The method according to claim 1, characterized in that The frame information further includes: a similarity value between the key frame of the target video and the similar key frame, wherein the similarity value is obtained by comparing image features of the key frame of the target video with image features of the similar key frame; The step of performing video matching comparison between any candidate video and the target video and determining a duplicate video corresponding to the target video from the candidate video set includes: Constructing a bipartite graph according to the key frame set of the target video and the key frame set of the candidate video; in the bipartite graph, there is a connecting edge between the key frame of the target video and the key frame of the candidate video, and the weight of the connecting edge represents the similarity between the frames, and the weight is determined based on the similarity value; Based on the bipartite graph, matching frame pairs between the key frame set of the target video and the key frame set of the candidate video are calculated, each matching frame pair including: a key frame of the target video and a similar key frame of the candidate video; If it is determined that the candidate video matches the target video according to the matching frame pair, then the candidate video and the target video are duplicate videos.

4. The method according to claim 3, characterized in that The frame information also includes: the total number of key frames in the candidate video; The determining, according to the matching frame pairs, that the candidate video matches the target video comprises: According to the matching frame pairs, at least a portion of the following matching information is obtained: the number of the matching frame pairs, the ratio of the key frames in the matching frame pairs to the total number of key frames in the target video to which they belong, the ratio of the similar key frames in the matching frame pairs to the total number of key frames in the candidate video to which they belong, and similarity information determined based on the similarity values ​​of the matching frame pairs; If the matching information between the candidate video and the target video meets the matching condition, it is determined that the candidate video matches the target video.

5. The method according to claim 1, characterized in that The method further comprises: Acquire a first video to be stored in the database; Extracting key frames of the first video and image features of each of the key frames; The image feature and frame information of the key frame corresponding to the image feature are stored in the database, wherein the frame information includes: a video identifier of the first video to which the key frame belongs.

6. The method according to claim 5, characterized in that The frame information of the key frame corresponding to the image feature also includes: The total number of key frames extracted from the first video.

7. A device for retrieving duplicate videos, characterized in that: The device is used to retrieve whether there is a duplicate video that is duplicated with the target video in the database; the device comprises: An information acquisition module, used to extract key frames included in the target video and image features of each key frame; The similar frame acquisition module is used to perform the following operations on each key frame of the target video: by comparing the image features of the key frame with the image features stored in the database, retrieve the frame information of the similar key frame similar to the key frame of the target video, and the frame information includes: the video identifier of the candidate video to which the similar key frame belongs; A video acquisition module, configured to aggregate similar key frames belonging to the same candidate video based on the video identifier to obtain a candidate video set, wherein the candidate video set includes at least one candidate video, and each candidate video includes at least one similar key frame; The video matching module is used to compare any candidate video with the target video for video matching, and determine the duplicate video corresponding to the target video from the candidate video set.

8. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor implements the method according to any one of claims 1 to 6 by running the executable instructions.

9. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When the instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images