Method and device for checking multimedia resources, electronic equipment and storage medium

Abstract

The present disclosure relates to a multimedia resource duplication checking method and device, electronic equipment and storage medium. The method comprises: obtaining a multimedia resource to be checked; performing image feature extraction on the image frames in the multimedia resource to be checked; searching for full-image candidate features corresponding to the image features from a first feature library, and searching for local-image candidate features corresponding to the image features from a second feature library; and determining a multimedia resource duplicated with the multimedia resource to be checked based on the searched full-image candidate features and local-image candidate features. The present disclosure can use the extracted full-image features and / or local-image features to cross-check in the full-image feature candidate library and the local-image feature candidate library, eliminate the interference of the size and position changes of the core picture area of the multimedia resource to be checked relative to the core picture area of the original multimedia resource on the duplication checking accuracy, and reduce the leakage rate of the multimedia resource duplication checking.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more specifically, to methods, apparatus, electronic devices, and storage media for detecting duplicate multimedia resources. Background Technology

[0002] Video plagiarism detection refers to using video search capabilities to retrieve videos with similar content from historical videos. It is a crucial means of maintaining a healthy short video ecosystem, primarily used to protect the legitimate rights and interests of original creators and combat plagiarism.

[0003] Cheating methods used to plagiarize video works include: scaling the original video, adjusting the screen orientation (landscape or portrait), adding borders, applying Gaussian blur to certain areas, and masking subtitles or watermarks. These editing operations alter the size and position of the core image area in the edited video. Related technologies often rely on extracting features from the entire video frame for plagiarism detection. However, when the core image area only occupies a portion of the entire video frame, the extracted features may not accurately represent this area. This could lead to instances where plagiarized videos with identical core image areas are mistakenly identified as plagiarized, resulting in a high rate of detection during plagiarism checks. Summary of the Invention

[0004] This disclosure provides a method, apparatus, electronic device, and storage medium for multimedia resource plagiarism detection, in order to at least solve the problem of high leakage rate in video plagiarism detection in the aforementioned related technologies.

[0005] According to a first aspect of the present disclosure, a method for detecting duplicate multimedia resources is provided, comprising: acquiring a multimedia resource to be detected; performing image feature extraction on image frames in the multimedia resource to be detected to obtain image features of the image frames; wherein the image features include full-image features and / or local image features of the image frames, the local image features being core motion region image features contained in the image frames; searching for full-image candidate features corresponding to the image features from a first feature library, and searching for local image candidate features corresponding to the image features from a second feature library; wherein the first feature library stores full-image candidate features of image frames of multiple first candidate multimedia resources, and the second feature library stores local image candidate features of image frames of multiple second candidate multimedia resources; and determining, based on the searched full-image candidate features and local image candidate features, multimedia resources that duplicate the multimedia resource to be detected among the multiple first candidate multimedia resources and the multiple second candidate multimedia resources.

[0006] Optionally, when the image frame includes a predetermined region, the image features include full-image features extracted from the entire image of the image frame, and local image features extracted from the region of the image frame other than the predetermined region, wherein the predetermined region is a non-core motion region of the image frame; when the image frame does not include the predetermined region, the image features include full-image features extracted from the entire image of the image frame.

[0007] Optionally, the step of searching for full-image candidate features corresponding to the image features from the first feature library and searching for local image candidate features corresponding to the image features from the second feature library includes: searching for full-image candidate features from the first feature library whose feature similarity to the image features reaches a preset threshold, and searching for local image candidate features from the second feature library whose feature similarity to the image features reaches the preset threshold.

[0008] Optionally, the step of determining the multimedia resource that duplicates the multimedia resource to be deduplicated from the plurality of first candidate multimedia resources and the plurality of second candidate multimedia resources based on the searched full-image candidate features and local image candidate features includes: finding a first target candidate multimedia resource to which the image frame corresponding to the searched full-image candidate features belongs among the plurality of first candidate multimedia resources; and finding a second target candidate multimedia resource to which the image frame corresponding to the searched local image candidate features belongs among the plurality of second candidate multimedia resources; and determining the multimedia resource that duplicates the multimedia resource to be deduplicated from the first target candidate multimedia resources and the second target candidate multimedia resources.

[0009] Optionally, determining the multimedia resources that duplicate the multimedia resource to be checked from the first target candidate multimedia resources and the second target candidate multimedia resources includes: if there are duplicate candidate multimedia resources in the first target candidate multimedia resources and the second target candidate multimedia resources, determining a specific candidate multimedia resource in the first target candidate multimedia resources and the second target candidate multimedia resources as a multimedia resource that duplicates the multimedia resource to be checked; wherein, the specific candidate multimedia resource includes candidate multimedia resources in the first target candidate multimedia resources and the second target candidate multimedia resources other than the duplicate candidate multimedia resources, and candidate multimedia resources in the duplicate candidate multimedia resources whose corresponding feature similarity satisfies a preset condition; wherein, the feature similarity corresponding to the duplicate candidate multimedia resource is the feature similarity between the image features of the image frame in the duplicate candidate multimedia resource and the image features of the image frame contained in the multimedia resource to be checked.

[0010] Optionally, there is at least one specific candidate multimedia resource. Determining that a specific candidate multimedia resource among the first target candidate multimedia resource and the second target candidate multimedia resource is a multimedia resource that duplicates the multimedia resource to be checked for duplication includes: for each specific candidate multimedia resource among the at least one specific candidate multimedia resource, performing the following operations: obtaining the image features of the image frames contained in the specific candidate multimedia resource and the feature similarity between them and the image features of the image frames contained in the multimedia resource to be checked for duplication; determining the feature similarity at the multimedia resource level between the specific candidate multimedia resource and the multimedia resource to be checked for duplication based on the feature similarity corresponding to the image frames contained in the specific candidate multimedia resource; determining that the specific candidate multimedia resource with the highest first number of feature similarities at the corresponding multimedia resource level among the at least one specific candidate multimedia resources is a multimedia resource that duplicates the multimedia resource to be checked for duplication.

[0011] Optionally, determining that the specific candidate multimedia resources with the highest feature similarity ranking among the at least one specific candidate multimedia resources are multimedia resources that duplicate the multimedia resource to be checked includes: for each specific candidate multimedia resource in the highest first number of positions, performing the following operations: obtaining the other feature similarity between other features of the specific candidate multimedia resource and other features of the multimedia resource to be checked, wherein the other features are features of different dimensions than image features; determining that the specific candidate multimedia resources with the highest other feature similarity ranking among the highest first number of positions are multimedia resources that duplicate the multimedia resource to be checked, wherein the first number is greater than the second number.

[0012] Optionally, the method further includes: when the image frame includes the predetermined region, storing the full-image features extracted from the entire image of the image frame into the first feature library, and storing the local image features extracted from the region of the image frame other than the predetermined region into the second feature library; when the image frame does not include the predetermined region, storing the full-image features extracted from the entire image of the image frame into the first feature library.

[0013] Optionally, after obtaining the multimedia resources to be checked for plagiarism, the method further includes: extracting a preset number of image frames to be checked from the multimedia resources to be checked for plagiarism; performing image feature extraction on the image frames in the multimedia resources to be checked for plagiarism to obtain the image features of the image frames, including: performing image feature extraction on the extracted preset number of image frames to be checked for plagiarism to obtain the image features of the image frames to be checked for plagiarism.

[0014] According to a second aspect of the present disclosure, a multimedia resource deduplication device is provided, comprising: an acquisition module configured to acquire multimedia resources to be deduplicated; a feature extraction module configured to perform image feature extraction on image frames in the multimedia resources to be deduplicated, to obtain image features of the image frames; wherein the image features include full-image features of the image frames and / or local image features, the local image features being core motion region image features contained in the image frames; a search module configured to search for full-image candidate features corresponding to the image features from a first feature library, and to search for local image candidate features corresponding to the image features from a second feature library; wherein the first feature library stores full-image candidate features of image frames of multiple first candidate multimedia resources, and the second feature library stores local image candidate features of image frames of multiple second candidate multimedia resources; and a determination module configured to determine, based on the searched full-image candidate features and local image candidate features, multimedia resources that duplicate the multimedia resources to be deduplicated from the multiple first candidate multimedia resources and the multiple second candidate multimedia resources.

[0015] Optionally, when the image frame includes a predetermined region, the image features include full-image features extracted from the entire image of the image frame, and local image features extracted from the region of the image frame other than the predetermined region, wherein the predetermined region is a non-core motion region of the image frame; when the image frame does not include the predetermined region, the image features include full-image features extracted from the entire image of the image frame.

[0016] Optionally, the search module is configured to: search for full-image candidate features from the first feature library that have a feature similarity to the image features that reaches a preset threshold, and search for local image candidate features from the second feature library that have a feature similarity to the image features that reaches the preset threshold.

[0017] Optionally, the determining module is configured to: find the first target candidate multimedia resource to which the image frame corresponding to the searched full-image candidate feature belongs among the plurality of first candidate multimedia resources; and find the second target candidate multimedia resource to which the image frame corresponding to the searched local image candidate feature belongs among the plurality of second candidate multimedia resources; and determine the multimedia resource that duplicates the multimedia resource to be checked from the first target candidate multimedia resource and the second target candidate multimedia resource.

[0018] Optionally, the determining module is configured to: when there are duplicate candidate multimedia resources among the first target candidate multimedia resources and the second target candidate multimedia resources, determine a specific candidate multimedia resource among the first target candidate multimedia resources and the second target candidate multimedia resources as a multimedia resource that duplicates the multimedia resource to be checked for duplication; wherein, the specific candidate multimedia resource includes candidate multimedia resources among the first target candidate multimedia resources and the second target candidate multimedia resources other than the duplicate candidate multimedia resources, and candidate multimedia resources among the duplicate candidate multimedia resources whose corresponding feature similarity satisfies a preset condition; wherein, the feature similarity corresponding to the duplicate candidate multimedia resource is the feature similarity between the image features of the image frames in the duplicate candidate multimedia resource and the image features of the image frames contained in the multimedia resource to be checked for duplication.

[0019] Optionally, the determining module is configured to: for each specific candidate multimedia resource among at least one specific candidate multimedia resource, perform the following operations: obtain the image features of the image frames contained in the specific candidate multimedia resource and the feature similarity between them and the image features of the image frames contained in the multimedia resource to be deduplicated; determine the multimedia resource-level feature similarity between the specific candidate multimedia resource and the multimedia resource to be deduplicated based on the feature similarity corresponding to the image frames contained in the specific candidate multimedia resource; determine that the specific candidate multimedia resource with the highest first-order feature similarity at the corresponding multimedia resource level among the at least one specific candidate multimedia resources is a multimedia resource that duplicates the multimedia resource to be deduplicated.

[0020] Optionally, the determining module is configured to: for each specific candidate multimedia resource in the first number of specific candidate multimedia resources ranked first, perform the following operations: obtain the similarity of other features between other features of the specific candidate multimedia resource and other features of the multimedia resource to be deduplicated, wherein the other features are features of different dimensions from image features; determine that the specific candidate multimedia resources in the first number of specific candidate multimedia resources ranked first, whose corresponding other feature similarity ranks first, are multimedia resources that duplicate the multimedia resource to be deduplicated, wherein the first number is greater than the second number.

[0021] Optionally, the multimedia resource deduplication device further includes: a first storage module configured to, when the image frame includes the predetermined region, store full-image features extracted from the entire image of the image frame into the first feature library, and store local image features extracted from the region of the image frame other than the predetermined region into the second feature library; and a second storage module configured to, when the image frame does not include the predetermined region, store full-image features extracted from the entire image of the image frame into the first feature library.

[0022] Optionally, the multimedia resource deduplication device further includes: an extraction module configured to extract a preset number of image frames to be deduplicated from the multimedia resource to be deduplicated; and a feature extraction module configured to perform image feature extraction on the extracted preset number of image frames to be deduplicated, thereby obtaining the image features of the image frames to be deduplicated.

[0023] According to a third aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to execute the instructions to implement a multimedia resource deduplication method according to the present disclosure.

[0024] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein when instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device is enabled to perform a multimedia resource deduplication method according to the present disclosure.

[0025] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements a multimedia resource deduplication method according to the present disclosure.

[0026] The technical solutions provided by the embodiments of this disclosure bring at least the following beneficial effects:

[0027] This disclosure can extract full-image features and / or local image features of the multimedia resource to be checked for plagiarism, and use the extracted full-image features and / or local image features to perform cross-search in the full-image feature candidate library and the local image feature candidate library. This can eliminate the interference on the plagiarism detection accuracy when the size and position of the core screen area of ​​the copied multimedia resource changes relative to the core screen area of ​​the original multimedia resource. In other words, it can identify plagiarism behavior in multimedia resource works with high accuracy and reduce the leakage rate of multimedia resource plagiarism detection.

[0028] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0029] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0030] Figure 1 This is a flowchart illustrating a multimedia resource deduplication method according to an exemplary embodiment of the present disclosure;

[0031] Figure 2 This is a schematic diagram showing a comparison between the results of the deduplication of multimedia resources in this disclosure and the results of deduplication of multimedia resources in related technologies;

[0032] Figure 3 This is a flowchart illustrating a specific implementation of a multimedia resource deduplication method according to an exemplary embodiment of the present disclosure;

[0033] Figure 4 This is a block diagram illustrating a multimedia resource deduplication device according to an exemplary embodiment of the present disclosure;

[0034] Figure 5 This is a block diagram illustrating an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0035] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0036] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following examples do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0037] It should be noted that the phrase "at least one of several items" in this disclosure refers to three parallel cases: "any one of the several items", "a combination of any number of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel cases: (1) including A; (2) including B; (3) including A and B. As another example, "performing at least one of step one and step two" indicates the following three parallel cases: (1) performing step one; (2) performing step two; (3) performing both step one and step two.

[0038] Video plagiarism detection refers to using video search capabilities to identify videos with similar content from a massive amount of historical videos. It is a crucial means of maintaining a healthy short video ecosystem, primarily used to protect the legitimate rights of original creators and combat plagiarism. In terms of content ecology, plagiarism detection identifies duplicate works and plagiarists, leading to varying degrees of penalties, including but not limited to stopping public traffic and banning accounts. Regarding user consumption, it influences the video recommendation system's distribution strategy by identifying perceived repetition, preventing users from repeatedly viewing similar videos within a short period.

[0039] With the continuous development of the short video trend, major video platforms have combined machine and human review to establish video plagiarism detection capabilities in order to combat content theft, encourage originality, and maintain and distribute high-quality videos. At the same time, content plagiarists are constantly increasing their investment and variety in their methods, attempting to use "deep editing" techniques—which are harder for machines to detect—to circumvent platform plagiarism checks and profit from them. Therefore, the types of repetitive scenarios that video plagiarism detection needs to address are constantly increasing, including but not limited to: identifying two videos that are wholly or partially identical, mutually inclusive, fast-forwarded, slow-motion, or reversed; and plagiarism behaviors such as adding, deleting, or modifying filters, effects, background music, images, text, borders, and split-screen effects.

[0040] In related technologies, the following problems mainly exist when performing video plagiarism checks:

[0041] 1. For cheating behaviors that are intended to maliciously bypass video plagiarism checks, the recall capability is limited.

[0042] Common cheating methods for video plagiarism include: scaling the original video, adjusting the screen orientation (landscape or portrait), adding borders, applying Gaussian blur to certain areas, and masking subtitles or watermarks. These editing operations alter the size and position of the core image area in the edited video. Related technologies often rely on extracting features from the entire video frame for plagiarism detection. However, when the core image area only occupies a portion of the entire video frame, the extracted features are insufficient to accurately represent it. This can lead to instances where plagiarized videos with identical core image areas are mistakenly identified as plagiarized, resulting in a high miss rate and hindering the ability of plagiarism detection systems to identify maliciously plagiarized videos.

[0043] 2. In order to improve retrieval recall capabilities, the index database was expanded blindly, but there was no clear basis for selecting index features.

[0044] A common way to improve the recall rate of video plagiarism detection systems is to add new feature retrieval libraries. However, videos have a wide variety of features, and blindly increasing the types of retrieval features will increase the storage and computing costs required for retrieval. In fact, some features are suitable as one source of retrieval, while others are suitable for filtering and verifying candidate results after retrieval. This requires experimental testing to confirm the selection criteria.

[0045] 3. When performing video plagiarism checks, the characteristics of purely visual capabilities are often insufficient to meet the actual plagiarism check requirements.

[0046] In the short video ecosystem, situations frequently arise where scenes are highly similar but the content differs: 1) Users often introduce trendy new features, such as templates, magic emojis, and co-op videos; 2) News, dance, singing, playing instruments, explanations, and variety shows are repeatedly filmed in the exact same setting; 3) Different people play the same game. Purely visual plagiarism detection capabilities cannot address these unique and common scenarios, leading to a high false positive rate in video plagiarism checks, which can dampen the enthusiasm of original creators.

[0047] 4. After obtaining similar videos, the logic of the plagiarism detection strategy is coupled in the core main process service of plagiarism detection and is not easy to extend.

[0048] After retrieving candidate duplicate video works using the search library, certain post-processing strategies are still needed for screening and judgment to improve the accuracy of video plagiarism detection. If the post-processing strategy is coupled into the core video plagiarism detection service, it will cause problems such as frequent upgrades of the core video plagiarism detection service, the inability to decouple the development of the core link from the iteration of the post-processing strategy, and the inability to assign tasks to different developers, making it difficult to expand the overall architecture of video plagiarism detection.

[0049] To address the aforementioned issues in related technologies, the multimedia resource plagiarism detection method disclosed herein can extract full-image features and / or local image features of the multimedia resource to be plagiarized. It then uses the extracted full-image features and / or local image features to perform cross-referencing within the full-image feature candidate library and the local image feature candidate library. This eliminates interference with plagiarism detection accuracy when the size and position of the core image area of ​​the copied multimedia resource changes relative to the core image area of ​​the original multimedia resource. In other words, it can accurately identify plagiarism in multimedia resource works and reduce the miss rate in multimedia resource plagiarism detection.

[0050] Figure 1 This is a flowchart illustrating a multimedia resource deduplication method according to an exemplary embodiment of the present disclosure.

[0051] Reference Figure 1In step 101, the multimedia resource to be checked for plagiarism can be obtained, that is, the multimedia resource whose plagiarism is to be determined can be obtained. For example, the multimedia resource in this disclosure can be video, audio and video, etc.

[0052] In step 102, image feature extraction can be performed on the image frames in the multimedia resource to be deduplicated, to obtain the image features of the image frames. These image features can include full-image features and / or local image features. Local image features can be the image features of the core motion region contained within the image frame; full-image features can be the image features corresponding to the entire image frame.

[0053] According to an exemplary embodiment of this disclosure, when an image frame includes a predetermined region, the image features may include full-image features extracted from the entire image frame, and local image features extracted from the region of the image frame excluding the predetermined region. The predetermined region may be a non-core motion region of the image frame. For example, the "predetermined region" may be the top and bottom border regions of the image frame, and the region within the image frame excluding the predetermined region is the core image region excluding the top and bottom border regions. This core image region may contain the content of the actual object being captured, such as people, animals, or scenery.

[0054] When an image frame does not include a predetermined region, the image features can include full-image features extracted from the entire image frame. That is, when the image frame does not include the top and bottom border regions, i.e., when the entire image frame is the core image region, the image features can include image features extracted from the entire core image region of the image frame. Thus, this disclosure can utilize the extracted full-image features and / or local image features to perform cross-searches in the full-image feature candidate library and the local image feature candidate library. This can eliminate interference with the accuracy of plagiarism detection when the size and position of the core image region of the copied multimedia resource changes relative to the core image region of the original multimedia resource, ensuring the accuracy of multimedia resource plagiarism detection.

[0055] According to an exemplary embodiment of this disclosure, a preset number of image frames to be checked can be extracted from the multimedia resources to be checked for plagiarism, and then image feature extraction can be performed on the extracted preset number of image frames to be checked to obtain the image features of the image frames to be checked for plagiarism.

[0056] The "preset number" can be set according to actual needs; for example, it can be set to 11. It should be noted that since a multimedia resource may contain many image frames, using all image frames of the multimedia resource to perform plagiarism checking would consume a significant amount of computing resources. Therefore, a preset number of image frames can be extracted from the multimedia resource to be checked, and then these extracted preset number of image frames can be used for plagiarism checking. This can significantly reduce the amount of computation in the multimedia resource plagiarism checking process and save considerable computing resources.

[0057] Furthermore, when extracting image frames to be checked from the multimedia resources to be checked for plagiarism, a preset number of image frames can be extracted evenly according to the length of the multimedia resources. For example, 11 image frames can be extracted evenly. Then, the image features of each of the 11 extracted image frames can be extracted. For example, a deep learning model can be used to extract the embedding feature vector of each of the 11 extracted image frames as the image features of that image frame.

[0058] It should be noted that in multimedia resource transfer scenarios, there are situations where only a small segment of the original multimedia resource is extracted. In this case, the image frames extracted using the uniform frame extraction scheme may exhibit frame errors, resulting in a very low similarity between the transferred multimedia resource and the original multimedia resource, leading to a certain degree of missed recall. Therefore, the uniform frame extraction scheme can be upgraded to a keyframe extraction scheme, which extracts image frames that show significant differences from the multimedia resource to be checked, i.e., it extracts representative image frames from the multimedia resource to be checked, and uses multiple image frames with significant differences between them for multimedia resource deduplication. This can effectively avoid the missed recall problem caused by the uniform frame extraction scheme and reduce the missed recall rate of multimedia resource deduplication.

[0059] In step 103, candidate full-image features corresponding to the image features can be searched from the first feature library, and candidate local image features corresponding to the image features can be searched from the second feature library. The first feature library can store candidate full-image features of multiple image frames of first candidate multimedia resources; the second feature library can store candidate local image features of multiple image frames of second candidate multimedia resources.

[0060] According to an exemplary embodiment of this disclosure, candidate features of the entire image that have a feature similarity to image features reaching a preset threshold can be searched from a first feature library, and candidate features of local images that have a feature similarity to image features reaching a preset threshold can be searched from a second feature library. The "preset threshold" can be set according to actual needs, and this disclosure does not specifically limit it. Thus, if the feature similarity between a candidate image feature and the image features of the image frame to be deduplicated reaches the preset threshold, it indicates that the corresponding candidate image frame is relatively similar to the image frame to be deduplicated. Therefore, the candidate multimedia resource to which the candidate image frame belongs is likely to be a multimedia resource that duplicates the multimedia resource to be deduplicated, ensuring the accuracy of multimedia resource deduplication.

[0061] In step 104, based on the searched full-image candidate features and local image candidate features, multimedia resources that duplicate the multimedia resource to be checked can be identified from multiple first candidate multimedia resources and multiple second candidate multimedia resources.

[0062] According to an exemplary embodiment of this disclosure, a first target candidate multimedia resource to which an image frame corresponding to a full-image candidate feature belongs can be found among a plurality of first candidate multimedia resources; and a second target candidate multimedia resource to which an image frame corresponding to a local image candidate feature belongs among a plurality of second candidate multimedia resources. Next, multimedia resources that duplicate the multimedia resource to be checked can be determined from the first and second target candidate multimedia resources.

[0063] For example, as mentioned earlier, 11 image frames can be extracted from the multimedia resource to be checked for plagiarism. For each of the 11 extracted image frames, a cross-referencing can be performed in the first feature library and the second feature library. In this way, each image frame may have at least one full-image candidate feature selected from the first feature library and at least one local image candidate feature selected from the second feature library. Since each full-image candidate feature corresponds to a candidate image frame, and this candidate image frame belongs to a candidate multimedia resource, each image frame may correspond to at least one first candidate multimedia resource.

[0064] Similarly, since each local image candidate feature corresponds to a candidate image frame, and this candidate image frame belongs to a candidate multimedia resource, each image frame to be checked for duplication may correspond to at least one second candidate multimedia resource. Next, from the at least one first candidate multimedia resource and at least one second candidate multimedia resource corresponding to the 11 extracted image frames to be checked for duplication, the multimedia resources that overlap with the multimedia resources to be checked for duplication can be determined.

[0065] Thus, if the full-image features or local image features of a certain image frame match the image features of the image frame to be checked for duplication, it indicates that the image frame is very similar to the image frame to be checked. Therefore, the multimedia resource to which the image frame belongs is highly likely to be a duplicate of the multimedia resource to be checked. Thus, by identifying the multimedia resources that duplicate the multimedia resource to be checked from the first target candidate multimedia resources to which the image frame corresponding to the full-image candidate features belongs, and the second target candidate multimedia resources to which the image frame corresponding to the local image candidate features belongs, the accuracy of multimedia resource duplication detection can be guaranteed.

[0066] According to an exemplary embodiment of this disclosure, when there are duplicate candidate multimedia resources in the first target candidate multimedia resource and the second target candidate multimedia resource, a specific candidate multimedia resource in the first target candidate multimedia resource and the second target candidate multimedia resource can be determined as a multimedia resource that duplicates the multimedia resource to be checked.

[0067] The term "specific candidate multimedia resources" can include candidate multimedia resources other than duplicate candidate multimedia resources from the first and second target candidate multimedia resources, as well as candidate multimedia resources whose corresponding feature similarity satisfies a preset condition from the duplicate candidate multimedia resources. "Duplicate candidate multimedia resources" can be at least two identical candidate multimedia resources; "the feature similarity corresponding to the duplicate candidate multimedia resources" can be the feature similarity between the image features of the image frames in the duplicate candidate multimedia resources and the image features of the image frames contained in the multimedia resource to be deduplicated; "the corresponding feature similarity satisfies the preset condition" can be "the corresponding feature similarity is the maximum feature similarity," that is, "candidate multimedia resources whose corresponding feature similarity satisfies the preset condition from the duplicate candidate multimedia resources" can be candidate multimedia resources with the maximum corresponding feature similarity among at least two identical candidate multimedia resources.

[0068] It should be noted that "at least two identical candidate multimedia resources" can be selected from the same feature candidate library or from different feature candidate libraries. For example, "at least two identical candidate multimedia resources" can be selected from the full-image feature candidate library, from the local image feature candidate library, or from both the full-image feature candidate library and the local image feature candidate library.

[0069] As mentioned earlier, each image frame to be checked for deduplication may correspond to at least one first candidate multimedia resource and at least one second candidate multimedia resource. Therefore, the 11 image frames extracted from the multimedia resources to be checked may correspond to multiple first candidate multimedia resources and multiple second candidate multimedia resources. Among these multiple first candidate multimedia resources and multiple second candidate multimedia resources, there are likely to be at least two candidate multimedia resources that are identical to each other, that is, there are likely to be at least two candidate multimedia resources with the same multimedia resource identification (ID). In this case, a deduplication operation of the candidate multimedia resources is required. That is, only the candidate multimedia resources with the highest feature similarity among the at least two identical candidate multimedia resources can be used for subsequent filtering and sorting, avoiding the situation of using the same candidate multimedia resource multiple times for filtering and sorting, thus avoiding the ineffective waste of computational resources.

[0070] According to an exemplary embodiment of this disclosure, at least one specific candidate multimedia resource may exist, and for each specific candidate multimedia resource among the at least one specific candidate multimedia resource, the following operations may be performed:

[0071] Image features of image frames contained in a specific candidate multimedia resource can be obtained, along with the feature similarity between these features and those of image frames contained in the multimedia resource to be checked for duplication. Then, based on the feature similarity of the image frames contained in the specific candidate multimedia resource, the multimedia resource-level feature similarity between the specific candidate multimedia resource and the multimedia resource to be checked for duplication can be determined. Next, it can be determined that the specific candidate multimedia resource with the highest corresponding multimedia resource-level feature similarity among at least one specific candidate multimedia resource is the multimedia resource that duplicates the multimedia resource to be checked for duplication.

[0072] For example, as mentioned earlier, suppose 11 image frames are extracted from the multimedia resource to be checked for plagiarism. In this case, the image features of the 11 image frames contained in the specific candidate multimedia resource can be obtained, and their feature similarity is compared with the image features of the 11 image frames contained in the multimedia resource to be checked. This allows us to obtain the feature similarity between the image features of the first frame extracted from the specific candidate multimedia resource and the image features of the first frame extracted from the multimedia resource to be checked; the image features of the second frame extracted from the specific candidate multimedia resource and the image features of the second frame extracted from the multimedia resource to be checked; and so on, until the image features of the eleventh frame extracted from the specific candidate multimedia resource and the image features of the eleventh frame extracted from the multimedia resource to be checked. Thus, a total of 11 frame-level feature similarities can be obtained.

[0073] Then, based on the feature similarity of each image frame among the 11 image frames extracted from the specific candidate multimedia resource, the feature similarity at the multimedia resource level can be determined. For example, the mean or sum of the 11 feature similarities corresponding to the 11 image frames extracted from the specific candidate multimedia resource can be calculated as the feature similarity at the multimedia resource level for the specific candidate multimedia resource. Next, at least one specific candidate multimedia resource whose feature similarity at the corresponding multimedia resource level ranks in the first order can be identified as a multimedia resource that duplicates the multimedia resource to be checked.

[0074] The "first quantity" can be set according to actual needs, for example, it can be set to 10, and this disclosure does not impose specific restrictions on it. In this way, feature similarity at the multimedia resource level can be determined based on feature similarity at the frame level, and further filtering can be performed on at least one specific candidate multimedia resource based on feature similarity at the multimedia resource level, thereby narrowing the scope of multimedia resource deduplication and making it easier to find candidate multimedia resources with high similarity to the multimedia resource to be deduplicated.

[0075] According to an exemplary embodiment of this disclosure, for each specific candidate multimedia resource among the top first-number-positions of specific candidate multimedia resources, the following operations can be performed: The similarity between other features of the specific candidate multimedia resource and other features of the multimedia resource to be deduplicated can be obtained. These other features can be features of different dimensions than image features. Next, it can be determined that among the top first-number-positions of specific candidate multimedia resources, the specific candidate multimedia resources whose corresponding other feature similarity ranks in the top second-number-positions are multimedia resources that duplicate the multimedia resource to be deduplicated. The first number can be greater than the second number. The "second number" can be set according to actual needs; for example, it can be set to 5, and this disclosure does not impose specific limitations on it.

[0076] It should be noted that the aforementioned "other features" may include, but are not limited to, at least one of the following: facial features, text features, speech content features, background audio features, work tag features, magic emoji image features, and station logo features. Among these, "station logo features" indicate which multimedia resource platform the corresponding multimedia resource was copied from. Generally, if the multimedia resource to be checked has a station logo feature, it can be considered that the multimedia resource to be checked is a copied multimedia resource, that is, it can be considered a non-original plagiarized work.

[0077] Thus, because multimedia resource deduplication schemes based solely on full-image and local image features of image frames consider only a single dimension, when faced with a large number of multimedia resources with highly similar scenes, truly duplicate multimedia resources may be hidden among them, leading to a failure to recall genuine duplicate multimedia resources and causing a certain degree of missed recall. For example, as mentioned earlier, in the multimedia resource ecosystem, situations often arise where scenes are highly similar but the content is different: 1) Users often have some trendy ways of playing, such as templates, magic emoticons, and co-creation; 2) Multimedia resources such as news, dance, singing, playing instruments, explanations, and variety shows are continuously filmed in the exact same scene; 3) Different people operate the same game. In these cases, purely visual deduplication capabilities cannot solve these special and common scenarios, resulting in a high false positive rate in multimedia resource deduplication, which can dampen the enthusiasm of original authors for creating multimedia resources.

[0078] Therefore, in addition to using full-image features and local image features of image frames for multimedia resource plagiarism detection, features from multiple other dimensions such as text and audio can also be considered for multimedia resource plagiarism detection. This can effectively reduce the false positive rate of multimedia resource plagiarism detection and avoid discouraging original authors from creating multimedia resources.

[0079] Furthermore, the aforementioned method of multimedia resource plagiarism detection based on "other features" can be called a post-processing strategy service. This service involves acquiring and using multimodal features, designing customized plagiarism rules, and using machine learning models to verify similarity. Moreover, this post-processing strategy service often presents a decision tree structure, relies on diverse features, and has complex and variable plagiarism rules. If the post-processing strategy service is coupled into the core multimedia resource plagiarism detection service, it will cause problems such as frequent upgrades to the core service, the inability to decouple the development of the core process from the iteration of the post-processing strategy, and the inability to assign tasks to different developers, making the overall architecture of multimedia resource plagiarism detection difficult to expand.

[0080] Therefore, in this disclosure, the post-processing strategy service for multimedia resource deduplication based on "other features" can be decoupled from the aforementioned main deduplication process service. That is, after the core deduplication service performs retrieval recall and image feature similarity ranking, it can send candidate duplicate "multimedia resource pairs"—i.e., the multimedia resources to be judged as duplicates and the candidate multimedia resources initially screened as duplicates—to the post-processing strategy service. The post-processing strategy service then performs subsequent duplicate multimedia resource judgment based on pre-set post-processing strategies and multimodal features. Finally, the deduplication result from the post-processing strategy service can be returned to the core deduplication process service. In this way, by decoupling the main deduplication process service and the post-processing strategy service at the service level, it is easier for developers of different technology stacks to be responsible for them, which can improve the stability of the main deduplication process service and the implementation efficiency of the post-processing strategy service, while reducing the risk and cost of frequent upgrades to the main deduplication process service.

[0081] To support the post-processing strategy service in flexibly accessing multimodal features of multimedia resources, defining complex strategy logic, and efficiently executing strategies, this disclosure utilizes a separate platform supporting strategy management and computation, including a strategy management platform and a strategy execution engine. The strategy management platform efficiently and visually supports feature access, strategy development, and management; the strategy execution engine can acquire features with high performance and execute various strategies according to a Directed Acyclic Graph (DAG) topology. For example, the strategy management platform can support various feature import methods such as remote dictionary services (Redis), distributed storage systems (HBase), remote procedure calls (RPC), and Java application programming interfaces (APIs); it supports configuring feature access using an online user interface (UI) page; and it supports developing strategies on the front end using the MVEL low-code scripting language. The strategy execution engine supports features dependency resolution, feature acquisition, and strategy expression execution.

[0082] According to exemplary embodiments of this disclosure, when an image frame includes a predetermined region, full-image features extracted from the entire image frame can be stored in a first feature library, and local image features extracted from regions of the image frame other than the predetermined region can be stored in a second feature library. That is, when an image frame includes top and bottom border regions, full-image features extracted from the entire image frame can be stored in the first feature library, and local image features extracted from the core image region of the image frame other than the top and bottom border regions can be stored in the second feature library.

[0083] If the image frame does not include a predetermined region, the full-image features extracted from the entire image frame can be stored in the first feature library. That is, if the image frame does not include the top and bottom border regions, the overall core image region features extracted from the entire image frame can be stored in the first feature library. In this way, the full-image features and / or local image features of the image frames of the multimedia resource to be checked for plagiarism can be stored in the corresponding feature library, and can be used as candidate image features for the next multimedia resource plagiarism check.

[0084] In this disclosure, in addition to using the visual embedding features extracted from the entire image as the retrieval recall source, a parallel recall source can be added. For this parallel recall source, an image analysis model for detecting core motion regions of an image can be used to detect the core motion regions of the input image frame. For example, methods such as inter-frame differencing, color matching, region feature matching, optical flow analysis, and key position matching can be used to achieve core motion region detection. Then, the detected core motion regions can be cropped, and the visual embedding features of the cropped core motion regions can be extracted. This visual embedding feature is then used to construct an index library as the aforementioned parallel recall source.

[0085] Experiments have demonstrated that core motion region detection is effective for multimedia resources undergoing plagiarism checks. For resources with detected core motion regions, cross-searching can be performed using core motion region features and full-image features in the full-image feature index and core region feature index, respectively. For resources without detected core motion regions, cross-searching can be performed using full-image features in both the full-image feature index and core region feature index. Compared to the strategy before introducing the core region feature index, on multimedia resource data of tens of millions, the plagiarism detection miss rate decreased from 7.09% to 3.96%, a reduction of 3.13%.

[0086] Figure 2 This is a schematic diagram illustrating a comparison between the results of the multimedia resource retrieval method disclosed in this disclosure and the results of multimedia resource retrieval methods in related technologies. Figure 2 In the diagram, the left image shows a frame of a multimedia resource to be checked for plagiarism, while the upper right image shows the recall results using a multimedia resource plagiarism detection scheme from related technologies. It can be seen that the related technologies' multimedia resource plagiarism detection scheme recalled image frames with identical content in the top and bottom border areas but different content in the core image area, resulting in "false positives," which could discourage original authors from creating multimedia resources. The lower right image shows the recall results using the multimedia resource plagiarism detection method of this disclosure. It can be seen that the multimedia resource plagiarism detection method of this disclosure recalled image frames with different content in the top and bottom border areas but identical content in the core image area, meaning that this disclosure has identified the truly plagiarized work.

[0087] Figure 3 This is a flowchart illustrating a specific implementation of a multimedia resource deduplication method according to an exemplary embodiment of the present disclosure. It mainly includes five steps: image feature extraction, dual-database retrieval and recall, filtering and sorting, verification judgment and post-processing of the deduplication system.

[0088] Reference Figure 3 In step 301, the videos to be checked for plagiarism can be obtained, i.e., videos whose plagiarism is to be determined. Each video can correspond to a video ID, i.e., a photo ID, to uniquely identify the corresponding video. The video IDs of different videos are different from each other.

[0089] In step 302, multiple video frames to be checked for plagiarism are extracted from the video to be checked. For example, 11 video frames can be extracted from the video to be checked. The video frame extraction method can be uniform frame extraction or keyframe extraction. The specific extraction process has been described in detail in the previous embodiment and will not be repeated here.

[0090] In step 303, the full-image features of each of the extracted multiple video frames to be checked for plagiarism are extracted, that is, the overall image features of each video frame to be checked for plagiarism are extracted. For example, a deep learning model can be used to extract the embedding feature vector of each of the 11 extracted video frames to be checked for plagiarism as the full-image features of that video frame.

[0091] In step 304, the full-image features of each of the 11 extracted video frames to be deduplicated are stored in HBase. It should be noted that HBase can store full-image features of video frames from a large number of candidate videos. These full-image features in HBase are mainly used for the subsequent "verification and judgment" process.

[0092] In step 305, feature encoding is performed on the full-image features of each of the 11 extracted video frames to be deduplicated.

[0093] In step 306, the feature-encoded full-image features of each of the 11 video frames to be deduplicated are stored in the index database, i.e., the full-image feature candidate database. It should be noted that the index database can store feature-encoded full-image features of video frames from a large number of candidate videos.

[0094] In step 307, a preset region detection can be performed on each of the 11 video frames to be checked for duplication, namely, the upper and lower border regions can be detected.

[0095] In step 308, the presence of a preset region in the video frame to be deduplicated is determined based on the preset region detection results.

[0096] In step 309, if the video frame to be checked for plagiarism has a preset area, the core image area of ​​the video frame to be checked for plagiarism is cropped out except for the preset area, that is, the core image area of ​​the video frame to be checked for plagiarism is cropped out except for the top and bottom border areas.

[0097] In step 3010, local image features of the core frame region cropped from the video frame to be checked for plagiarism are extracted and stored in HBase. It should be noted that HBase can store local image features of video frames from a large number of candidate videos; these local image features in HBase are mainly used in the subsequent "verification and judgment" process.

[0098] In step 3011, feature encoding is performed on the local image features of the video frame to be deduplicated, and the feature-encoded local image features of the video frame to be deduplicated are stored in an index library, i.e., a local image feature candidate library. It should be noted that the index library can store feature-encoded local image features of video frames from a large number of candidate videos.

[0099] In step 3012, the feature-encoded local image features of each of the 11 video frames to be deduplicated are used to perform retrieval and recall in the retrieval database. The "retrieval database" is used to perform retrieval and recall by utilizing a large number of feature-encoded local image features stored in the index database.

[0100] In step 3013, the feature-encoded local image features of each of the 11 video frames to be deduplicated are used to perform a retrieval and recall operation in the retrieval database. The "retrieval database" utilizes a large number of feature-encoded full-image features stored in the index database for retrieval and recall.

[0101] In step 3014, if a preset region exists in the video frame to be checked for plagiarism, the full-image features of the video frame to be checked, after feature encoding, are retrieved in the retrieval database.

[0102] In step 3015, if a preset region exists in the video frame to be checked for plagiarism, the full-image features of the video frame to be checked, after feature encoding, are retrieved in the retrieval database.

[0103] In step 3016, if a preset region exists in the video frame to be deduplicated, multiple candidate videos contained in the retrieval results of the small and large search databases are merged and deduplicated: for at least two candidate videos that are identical to each other, only the candidate video with the highest corresponding feature similarity is used for subsequent filtering and sorting.

[0104] In step 3017, if there is no preset region in the video frame to be checked for plagiarism, the full-image features of the video frame to be checked, after feature encoding, are retrieved in the retrieval database.

[0105] In step 3018, if there is no preset region in the video frame to be checked for plagiarism, the full-image features of the video frame to be checked, after feature encoding, are retrieved in the retrieval database.

[0106] In step 3019, if there is no preset region in the video frame to be deduplicated, multiple candidate videos contained in the retrieval results of the small and large search databases are merged and deduplicated: for at least two candidate videos that are the same to each other, only the candidate video with the highest feature similarity is used for subsequent filtering and sorting.

[0107] In step 3020, based on the feature similarity of each candidate video among the multiple candidate videos, the top 50 candidate videos with the highest feature similarity are selected from the multiple candidate videos. It should be noted that the "feature similarity of each candidate video" here refers to the feature similarity of the video frames contained in that candidate video; that is, the feature similarity of the video frames is used here to represent the feature similarity of the candidate video to which that video frame belongs.

[0108] In step 3021, for each of the 50 candidate videos, the video-level feature similarity of the candidate video is determined based on the frame-level feature similarity, and based on the video-level feature similarity of each of the 50 candidate videos, the top 10 candidate videos with the highest video-level feature similarity are selected from the 50 candidate videos.

[0109] In step 3022, for each of the top 10 candidate videos with the highest video-level feature similarity, the full-image features and local image features of that candidate video are read from HBase. For each pair of works consisting of a candidate video and the video to be checked, the similarity between the candidate video and the video to be checked can be verified based on the full-image features of the candidate video, the local image features of the candidate video, the full-image features of the video to be checked, and the local image features of the video to be checked, using a combination of Convolutional Neural Network (CNN) and Oriented Fast and Rotated BRIEF (ORB) image algorithms.

[0110] In step 3023, further deduplication post-processing is performed through the post-processing strategy service. For example, for each pair of works, features from other dimensions such as facial features, text features, voice content features, background audio features, work tag features, magic expression image features, and station logo features are used to determine whether the candidate videos and the videos to be checked for deduplication contained in the work pair are duplicates, thus generating the final deduplication retrieval result of the entire video deduplication system.

[0111] In step 3024, the final deduplication result of the video to be checked is reported.

[0112] The multimedia resource plagiarism detection method proposed in this disclosure can extract full-image features and / or local image features of the video frames to be checked for plagiarism, and then perform cross-searching in the full-image feature candidate library and the local image feature candidate library using the extracted full-image features and / or local image feature candidate library. Then, CNN and ORB can be used to further verify and filter the cross-search results. Next, with the help of post-processing strategy services, multimodal features can be used to further filter the candidate videos obtained after verification and filtering, and finally, based on the filtering results, it can be determined whether the video to be checked is a plagiarized work. With tens of millions of data points, the entire plagiarism detection process can be completed in an average of 12 seconds, with a plagiarism detection accuracy of 97% and a miss rate of only 4%. The miss rate refers to the proportion of videos that are actually duplicated among those judged as non-duplicate by the machine.

[0113] Figure 4 This is a block diagram illustrating a multimedia resource deduplication device according to an exemplary embodiment of the present disclosure.

[0114] Reference Figure 4 The multimedia resource plagiarism detection device 400 may include an acquisition module 401, a feature extraction module 402, a search module 403, and a determination module 404.

[0115] The acquisition module 401 can acquire the multimedia resources to be checked for plagiarism, that is, acquire the multimedia resources for which it is determined whether they are plagiarized works. For example, the multimedia resources in this disclosure can be videos, audio and video, etc.

[0116] The feature extraction module 402 can perform image feature extraction on image frames in the multimedia resource to be checked for plagiarism, and obtain the image features of the image frames. These image features can include full-image features and / or local image features of the image frame. Local image features can be the image features of the core motion region contained in the image frame; full-image features can be the image features corresponding to the entire image frame.

[0117] According to an exemplary embodiment of this disclosure, when an image frame includes a predetermined region, the image features may include full-image features extracted from the entire image frame, and local image features extracted from the region of the image frame excluding the predetermined region. The predetermined region may be a non-core motion region of the image frame. For example, the "predetermined region" may be the top and bottom border regions of the image frame, and the region within the image frame excluding the predetermined region is the core image region excluding the top and bottom border regions. This core image region may contain the content of the actual object being captured, such as people, animals, or scenery.

[0118] When an image frame does not include a predetermined region, the image features can include full-image features extracted from the entire image frame. That is, when the image frame does not include the top and bottom border regions, i.e., when the entire image frame is the core image region, the image features can include image features extracted from the entire core image region of the image frame. Thus, this disclosure can utilize the extracted full-image features and / or local image features to perform cross-searches in the full-image feature candidate library and the local image feature candidate library. This can eliminate interference with the accuracy of plagiarism detection when the size and position of the core image region of the copied multimedia resource changes relative to the core image region of the original multimedia resource, ensuring the accuracy of multimedia resource plagiarism detection.

[0119] According to an exemplary embodiment of this disclosure, the multimedia resource plagiarism detection device 400 may further include an extraction module. This extraction module can extract a preset number of image frames to be checked from the multimedia resources to be checked, and then the feature extraction module 402 can perform image feature extraction on the extracted preset number of image frames to obtain the image features of the image frames to be checked.

[0120] The "preset number" can be set according to actual needs; for example, it can be set to 11. It should be noted that since a multimedia resource may contain many image frames, using all image frames of the multimedia resource to perform plagiarism checking would consume a significant amount of computing resources. Therefore, a preset number of image frames can be extracted from the multimedia resource to be checked, and then these extracted preset number of image frames can be used for plagiarism checking. This can significantly reduce the amount of computation in the multimedia resource plagiarism checking process and save considerable computing resources.

[0121] Furthermore, when extracting image frames to be checked from the multimedia resources to be checked for plagiarism, a preset number of image frames can be extracted evenly according to the length of the multimedia resources. For example, 11 image frames can be extracted evenly. Then, the image features of each of the 11 extracted image frames can be extracted. For example, a deep learning model can be used to extract the embedding feature vector of each of the 11 extracted image frames as the image features of that image frame.

[0122] It should be noted that in multimedia resource transfer scenarios, there are situations where only a small segment of the original multimedia resource is extracted. In this case, the image frames extracted using the uniform frame extraction scheme may exhibit frame errors, resulting in a very low similarity between the transferred multimedia resource and the original multimedia resource, leading to a certain degree of missed recall. Therefore, the uniform frame extraction scheme can be upgraded to a keyframe extraction scheme, which extracts image frames that show significant differences from the multimedia resource to be checked, i.e., it extracts representative image frames from the multimedia resource to be checked, and uses multiple image frames with significant differences between them for multimedia resource deduplication. This can effectively avoid the missed recall problem caused by the uniform frame extraction scheme and reduce the missed recall rate of multimedia resource deduplication.

[0123] The search module 403 can search for full-image candidate features corresponding to image features from the first feature library, and can search for local image candidate features corresponding to image features from the second feature library. The first feature library can store full-image candidate features of multiple first candidate multimedia resource image frames; the second feature library can store local image candidate features of multiple second candidate multimedia resource image frames.

[0124] According to an exemplary embodiment of this disclosure, the search module 403 can search for candidate features of the entire image that have a feature similarity to the image features reaching a preset threshold from the first feature library, and can search for candidate features of the local image that have a feature similarity to the image features reaching a preset threshold from the second feature library. The "preset threshold" can be set according to actual needs, and this disclosure does not specifically limit it. Thus, if the feature similarity between the candidate image features and the image features of the image frame to be checked reaches the preset threshold, it indicates that the corresponding candidate image frame is relatively similar to the image frame to be checked. Therefore, the candidate multimedia resource to which the candidate image frame belongs is likely to be a multimedia resource that overlaps with the multimedia resource to be checked, ensuring the accuracy of multimedia resource deduplication.

[0125] The determination module 404 can determine the multimedia resources that are duplicated with the multimedia resource to be checked from among multiple first candidate multimedia resources and multiple second candidate multimedia resources, based on the searched full-image candidate features and local image candidate features.

[0126] According to an exemplary embodiment of this disclosure, the determining module 404 can search for a first target candidate multimedia resource among a plurality of first candidate multimedia resources, to which the image frame corresponding to the full-image candidate feature belongs; and can search for a second target candidate multimedia resource among a plurality of second candidate multimedia resources, to which the image frame corresponding to the local image candidate feature belongs. Next, the determining module 404 can determine multimedia resources that duplicate the multimedia resource to be checked from the first target candidate multimedia resources and the second target candidate multimedia resources.

[0127] Thus, if the full-image features or local image features of a certain image frame match the image features of the image frame to be checked for duplication, it indicates that the image frame is very similar to the image frame to be checked. Therefore, the multimedia resource to which the image frame belongs is highly likely to be a duplicate of the multimedia resource to be checked. Thus, by identifying the multimedia resources that duplicate the multimedia resource to be checked from the first target candidate multimedia resources to which the image frame corresponding to the full-image candidate features belongs, and the second target candidate multimedia resources to which the image frame corresponding to the local image candidate features belongs, the accuracy of multimedia resource duplication detection can be guaranteed.

[0128] According to an exemplary embodiment of this disclosure, when there are duplicate candidate multimedia resources in the first target candidate multimedia resources and the second target candidate multimedia resources, the determining module 404 can determine that a specific candidate multimedia resource in the first target candidate multimedia resources and the second target candidate multimedia resources is a multimedia resource that duplicates the multimedia resource to be checked.

[0129] The term "specific candidate multimedia resources" can include candidate multimedia resources other than duplicate candidate multimedia resources from the first and second target candidate multimedia resources, as well as candidate multimedia resources whose corresponding feature similarity satisfies a preset condition from the duplicate candidate multimedia resources. "Duplicate candidate multimedia resources" can be at least two identical candidate multimedia resources; "the feature similarity corresponding to the duplicate candidate multimedia resources" can be the feature similarity between the image features of the image frames in the duplicate candidate multimedia resources and the image features of the image frames contained in the multimedia resource to be deduplicated; "the corresponding feature similarity satisfies the preset condition" can be "the corresponding feature similarity is the maximum feature similarity," that is, "candidate multimedia resources whose corresponding feature similarity satisfies the preset condition from the duplicate candidate multimedia resources" can be candidate multimedia resources with the maximum corresponding feature similarity among at least two identical candidate multimedia resources.

[0130] It should be noted that "at least two identical candidate multimedia resources" can be selected from the same feature candidate library or from different feature candidate libraries. For example, "at least two identical candidate multimedia resources" can be selected from the full-image feature candidate library, from the local image feature candidate library, or from both the full-image feature candidate library and the local image feature candidate library.

[0131] As mentioned earlier, each image frame to be checked for deduplication may correspond to at least one first candidate multimedia resource and at least one second candidate multimedia resource. Therefore, the 11 image frames extracted from the multimedia resources to be checked may correspond to multiple first candidate multimedia resources and multiple second candidate multimedia resources. Among these multiple first candidate multimedia resources and multiple second candidate multimedia resources, there are likely to be at least two candidate multimedia resources that are identical to each other, that is, there are likely to be at least two candidate multimedia resources with the same multimedia resource identification (ID). In this case, a deduplication operation of the candidate multimedia resources is required. That is, only the candidate multimedia resources with the highest feature similarity among the at least two identical candidate multimedia resources can be used for subsequent filtering and sorting, avoiding the situation of using the same candidate multimedia resource multiple times for filtering and sorting, thus avoiding the ineffective waste of computational resources.

[0132] According to an exemplary embodiment of this disclosure, at least one specific candidate multimedia resource may exist, and for each specific candidate multimedia resource among the at least one specific candidate multimedia resource, the determining module 404 may perform the following operations:

[0133] The determining module 404 can acquire the image features of image frames contained in a specific candidate multimedia resource and the feature similarity between these features and the image features of image frames contained in the multimedia resource to be deduplicated. Then, based on the feature similarity of the image frames contained in the specific candidate multimedia resource, the determining module 404 can determine the multimedia resource-level feature similarity between the specific candidate multimedia resource and the multimedia resource to be deduplicated. Next, the determining module 404 can determine that the specific candidate multimedia resource with the highest corresponding multimedia resource-level feature similarity among at least one specific candidate multimedia resource is a multimedia resource that duplicates the multimedia resource to be deduplicated.

[0134] The "first quantity" can be set according to actual needs, for example, it can be set to 10, and this disclosure does not impose specific restrictions on it. In this way, feature similarity at the multimedia resource level can be determined based on feature similarity at the frame level, and further filtering can be performed on at least one specific candidate multimedia resource based on feature similarity at the multimedia resource level, thereby narrowing the scope of multimedia resource deduplication and making it easier to find candidate multimedia resources with high similarity to the multimedia resource to be deduplicated.

[0135] According to an exemplary embodiment of this disclosure, for each specific candidate multimedia resource among the top first-number-position specific candidate multimedia resources, the determining module 404 may perform the following operations: The determining module 404 may obtain the similarity of other features between other features of the specific candidate multimedia resource and other features of the multimedia resource to be deduplicated. Wherein, other features may be features of different dimensions than image features. Next, the determining module 404 may determine that among the top first-number-position specific candidate multimedia resources, the specific candidate multimedia resources whose corresponding other feature similarity ranks in the top second-number-position are multimedia resources that duplicate the multimedia resource to be deduplicated. The first number may be greater than the second number. The "second number" can be set according to actual needs, for example, it may be set to 5, and this disclosure does not impose specific limitations on it.

[0136] It should be noted that the aforementioned "other features" may include, but are not limited to, at least one of the following: facial features, text features, speech content features, background audio features, work tag features, magic emoji image features, and station logo features. Among these, "station logo features" indicate which multimedia resource platform the corresponding multimedia resource was copied from. Generally, if the multimedia resource to be checked has a station logo feature, it can be considered that the multimedia resource to be checked is a copied multimedia resource, that is, it can be considered a non-original plagiarized work.

[0137] Thus, because multimedia resource deduplication schemes based solely on full-image and local image features of image frames consider only a single dimension, when faced with a large number of multimedia resources with highly similar scenes, truly duplicate multimedia resources may be hidden among them, leading to a failure to recall genuine duplicate multimedia resources and causing a certain degree of missed recall. For example, as mentioned earlier, in the multimedia resource ecosystem, situations often arise where scenes are highly similar but the content is different: 1) Users often have some trendy ways of playing, such as templates, magic emoticons, and co-creation; 2) Multimedia resources such as news, dance, singing, playing instruments, explanations, and variety shows are continuously filmed in the exact same scene; 3) Different people operate the same game. In these cases, purely visual deduplication capabilities cannot solve these special and common scenarios, resulting in a high false positive rate in multimedia resource deduplication, which can dampen the enthusiasm of original authors for creating multimedia resources.

[0138] Therefore, in addition to using full-image features and local image features of image frames for multimedia resource plagiarism detection, features from multiple other dimensions such as text and audio can also be considered for multimedia resource plagiarism detection. This can effectively reduce the false positive rate of multimedia resource plagiarism detection and avoid discouraging original authors from creating multimedia resources.

[0139] According to exemplary embodiments of this disclosure, the multimedia resource deduplication device 400 may further include a first storage module and a second storage module. When an image frame includes a predetermined region, the first storage module can store full-image features extracted from the entire image frame into a first feature library, and can store local image features extracted from regions of the image frame other than the predetermined region into a second feature library. That is, when an image frame includes top and bottom border regions, the first storage module can store full-image features extracted from the entire image frame into the first feature library, and can store local image features extracted from the core image region of the image frame other than the top and bottom border regions into the second feature library.

[0140] If the image frame does not include a predetermined region, the second storage module can store the full-image features extracted from the entire image frame into the first feature library. That is, if the image frame does not include the top and bottom border regions, the second storage module can store the overall core image region features extracted from the entire image frame into the first feature library. In this way, the full-image features and / or local image features of the image frames of the multimedia resource to be checked for plagiarism can be stored in the corresponding feature library, and can be used as candidate image features for the next multimedia resource plagiarism check.

[0141] Figure 5 This is a block diagram illustrating an electronic device 500 according to an exemplary embodiment of the present disclosure.

[0142] Reference Figure 5 The electronic device 500 includes at least one memory 501 and at least one processor 502. The at least one memory 501 stores instructions that, when executed by the at least one processor 502, perform a multimedia resource deduplication method according to an exemplary embodiment of the present disclosure.

[0143] As an example, electronic device 500 may be a PC, tablet, personal digital assistant, smartphone, or other device capable of executing the aforementioned instructions. Here, electronic device 500 is not necessarily a single electronic device, but may be a collection of any devices or circuits capable of executing the aforementioned instructions (or instruction sets) individually or in combination. Electronic device 500 may also be part of an integrated control system or system manager, or may be configured to interconnect with a portable electronic device locally or remotely (e.g., via wireless transmission) through an interface.

[0144] In electronic device 500, processor 502 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processor may also include analog processors, digital processors, microprocessors, multi-core processors, processor arrays, network processors, etc.

[0145] The processor 502 can execute instructions or code stored in the memory 501, which can also store data. Instructions and data can also be sent and received over a network via a network interface device, which can employ any known transmission protocol.

[0146] The memory 501 may be integrated with the processor 502, for example, by arranging RAM or flash memory within an integrated circuit microprocessor. Alternatively, the memory 501 may include a separate device, such as an external disk drive, a storage array, or other storage device usable by any database system. The memory 501 and the processor 502 may be operatively coupled, or may communicate with each other, for example, via I / O ports, network connections, etc., enabling the processor 502 to read files stored in the memory.

[0147] In addition, electronic device 500 may also include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, mouse, touch input device, etc.). All components of electronic device 500 can be interconnected via a bus and / or network.

[0148] According to exemplary embodiments of this disclosure, a computer-readable storage medium may also be provided, which, when executed by a processor of an electronic device, enables the electronic device to perform the aforementioned multimedia resource deduplication method. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage, hard disk drive (HDD), solid-state drive (SSD), card storage (such as multimedia cards, secure digital (SD) cards, or ultra-fast digital (XD) cards), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid-state drive, and any other device configured to store a computer program and any associated data, data files, and data structures in a non-transitory manner and to provide the computer program and any associated data, data files, and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the aforementioned computer-readable storage medium can run in an environment deployed in computer devices such as clients, hosts, agent devices, servers, etc. Furthermore, in one example, the computer program and any associated data, data files, and data structures are distributed across a networked computer system, such that the computer program and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner through one or more processors or computers.

[0149] According to exemplary embodiments of the present disclosure, a computer program product may also be provided, including a computer program that, when executed by a processor, implements the multimedia resource deduplication method according to the present disclosure.

[0150] According to the multimedia resource plagiarism detection method, apparatus, electronic device, and storage medium disclosed herein, full-image features and / or local image features of the multimedia resource to be checked can be extracted. The extracted full-image features and / or local image features are then used to perform cross-searches in the full-image feature candidate library and the local image feature candidate library. This can eliminate interference with the plagiarism detection accuracy when the size and position of the core image area of ​​the copied multimedia resource changes relative to the core image area of ​​the original multimedia resource. In other words, it can identify plagiarism in multimedia resource works with high accuracy and reduce the leakage rate of multimedia resource plagiarism detection.

[0151] Furthermore, since a multimedia resource may contain many image frames, using all image frames of the multimedia resource to perform deduplication would consume a significant amount of computational resources. Therefore, a predetermined number of image frames can be extracted from the multimedia resource to be deduplicated, and then these extracted image frames can be used for deduplication. This can significantly reduce the computational load during the deduplication process and save considerable computational resources.

[0152] Furthermore, if the feature similarity between a candidate image feature and the image feature of the image frame to be checked reaches a preset threshold, it means that the corresponding candidate image frame is similar to the image frame to be checked. In this case, the candidate multimedia resource to which the candidate image frame belongs is likely to be a multimedia resource that overlaps with the multimedia resource to be checked, which can ensure the accuracy of multimedia resource deduplication.

[0153] Furthermore, if the full-image features or local image features of a certain image frame match the image features of the image frame to be checked for duplication, it indicates that the image frame is very similar to the image frame to be checked. Therefore, the multimedia resource to which the image frame belongs is highly likely to be a duplicate of the multimedia resource to be checked. Thus, by identifying the multimedia resources that duplicate the multimedia resource to be checked from the first target candidate multimedia resources to which the image frame corresponding to the full-image candidate features belongs, and the second target candidate multimedia resources to which the image frame corresponding to the local image candidate features belongs, the accuracy of multimedia resource duplication detection can be guaranteed.

[0154] Furthermore, it is necessary to perform deduplication of candidate multimedia resources. That is, only the candidate multimedia resources with the highest feature similarity among at least two identical candidate multimedia resources can be used for subsequent screening and sorting. This avoids the situation of using the same candidate multimedia resource for screening and sorting multiple times, and can avoid the ineffective waste of computing resources.

[0155] Furthermore, feature similarity at the multimedia resource level can be determined based on frame-level feature similarity, and then further filtered in at least one specific candidate multimedia resource based on the feature similarity at the multimedia resource level. This can narrow down the scope of multimedia resource duplication checking and make it easier to find candidate multimedia resources with high similarity to the multimedia resource to be checked.

[0156] Furthermore, because multimedia resource deduplication schemes based solely on full-image and local image features of image frames consider only a single dimension, when faced with a large number of multimedia resources that are highly similar in scene to each other, truly duplicate multimedia resources may be hidden among them, leading to a failure to recall genuine duplicate resources and causing a certain number of missed recalls. This can also result in a high false positive rate, which can dampen the enthusiasm of original authors for creating multimedia resources. Therefore, in addition to using full-image and local image features of image frames for multimedia resource deduplication, it is also possible to comprehensively consider features from multiple other dimensions such as text and audio dimensions. This can effectively reduce the false positive rate of multimedia resource deduplication and avoid discouraging original authors from creating multimedia resources.

[0157] Furthermore, by decoupling the main plagiarism detection service and the post-processing strategy service at the service level, it is easier for developers of different technology stacks to be responsible for them respectively. This can improve the stability of the main plagiarism detection service and the implementation efficiency of the post-processing strategy service, while reducing the risk and cost of high-frequency upgrades to the main plagiarism detection service.

[0158] Furthermore, the full-image features and / or local image features of the image frames of the multimedia resource to be checked for plagiarism can be stored in the corresponding feature library, which can be used as candidate image features for the next multimedia resource plagiarism check.

[0159] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0160] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A method for detecting duplicate content in multimedia resources, characterized in that, include: Obtain the multimedia resources to be checked for plagiarism. Image feature extraction is performed on the image frames in the multimedia resource to be deduplicated to obtain the image features of the image frames; wherein, the image features include the full-image features of the image frame and / or local image features, the local image features are the core motion region image features contained in the image frame, and the core motion region is the region in the image frame containing the actual object being photographed; Search for full-image candidate features corresponding to the image features in the first feature library, and search for local image candidate features corresponding to the image features in the second feature library; wherein, the first feature library stores full-image candidate features of image frames of multiple first candidate multimedia resources, and the second feature library stores local image candidate features of image frames of multiple second candidate multimedia resources. Based on the searched full-image candidate features and local image candidate features, multimedia resources that duplicate the multimedia resource to be checked are identified among the plurality of first candidate multimedia resources and the plurality of second candidate multimedia resources. The step of searching for candidate full-image features corresponding to the image features from the first feature library and searching for candidate local image features corresponding to the image features from the second feature library includes: From the first feature library, search for full-image candidate features whose feature similarity to the image features reaches a preset threshold, and from the second feature library, search for local image candidate features whose feature similarity to the image features reaches the preset threshold.

2. The multimedia resource deduplication method as described in claim 1, characterized in that, When the image frame includes a predetermined region, the image features include full-image features extracted from the entire image of the image frame, and local image features extracted from the region of the image frame other than the predetermined region, wherein the predetermined region is a non-core motion region of the image frame. If the image frame does not include the predetermined region, the image features include full-image image features extracted from the full image of the image frame.

3. The multimedia resource deduplication method as described in claim 1, characterized in that, The step of determining multimedia resources that duplicate the multimedia resource to be deduplicated from the plurality of first candidate multimedia resources and the plurality of second candidate multimedia resources based on the searched full-image candidate features and local image candidate features includes: Find the first target candidate multimedia resource to which the image frame corresponding to the searched full-image candidate features belongs among the plurality of first candidate multimedia resources; and find the second target candidate multimedia resource to which the image frame corresponding to the searched local image candidate features belongs among the plurality of second candidate multimedia resources; From the first target candidate multimedia resource and the second target candidate multimedia resource, determine the multimedia resource that duplicates the multimedia resource to be checked.

4. The multimedia resource deduplication method as described in claim 3, characterized in that, The step of determining the multimedia resources that duplicate the multimedia resource to be checked from the first target candidate multimedia resources and the second target candidate multimedia resources includes: If there are duplicate candidate multimedia resources in the first target candidate multimedia resources and the second target candidate multimedia resources, a specific candidate multimedia resource in the first target candidate multimedia resources and the second target candidate multimedia resources is determined to be a multimedia resource that duplicates the multimedia resource to be checked for duplicates. The specific candidate multimedia resources include candidate multimedia resources other than the duplicate candidate multimedia resources in the first target candidate multimedia resources and the second target candidate multimedia resources, as well as candidate multimedia resources whose corresponding feature similarity satisfies a preset condition in the duplicate candidate multimedia resources; wherein, the feature similarity corresponding to the duplicate candidate multimedia resources is the feature similarity between the image features of the image frames in the duplicate candidate multimedia resources and the image features of the image frames contained in the multimedia resources to be deduplicated.

5. The multimedia resource deduplication method as described in claim 4, characterized in that, At least one of the specific candidate multimedia resources exists, and determining that the specific candidate multimedia resource among the first target candidate multimedia resource and the second target candidate multimedia resource is a multimedia resource that duplicates the multimedia resource to be checked for duplication includes: For each specific candidate multimedia resource in at least one specific candidate multimedia resource, perform the following operation: Obtain the image features of the image frames contained in the specific candidate multimedia resource and the feature similarity between them and the image features of the image frames contained in the multimedia resource to be deduplicated; Based on the feature similarity of the image frames contained in the specific candidate multimedia resource, the feature similarity at the multimedia resource level between the specific candidate multimedia resource and the multimedia resource to be deduplicated is determined. Among the at least one specific candidate multimedia resources, the specific candidate multimedia resource whose feature similarity at the corresponding multimedia resource level ranks first in the first number of positions is identified as a multimedia resource that duplicates the multimedia resource to be checked for duplication.

6. The multimedia resource deduplication method as described in claim 5, characterized in that, The step of determining the specific candidate multimedia resource that ranks first in the first number of times in the feature similarity at the corresponding multimedia resource level among the at least one specific candidate multimedia resource is a multimedia resource that duplicates the multimedia resource to be checked for duplication, includes: For each specific candidate multimedia resource in the first-ranked specific candidate multimedia resource, the following operations are performed: obtain the other feature similarity between other features of the specific candidate multimedia resource and other features of the multimedia resource to be deduplicated, wherein the other features are features of different dimensions from image features; Among the specific candidate multimedia resources ranked in the first number of positions, the specific candidate multimedia resources whose other feature similarity ranks in the second number of positions are determined to be multimedia resources that duplicate the multimedia resource to be checked, wherein the first number is greater than the second number.

7. The multimedia resource deduplication method as described in claim 2, characterized in that, Also includes: When the image frame includes the predetermined region, the full-image features extracted from the entire image of the image frame are stored in the first feature library, and the local image features extracted from the region of the image frame other than the predetermined region are stored in the second feature library. If the image frame does not include the predetermined region, the full-image features extracted from the entire image frame are stored in the first feature library.

8. The multimedia resource deduplication method as described in any one of claims 1 to 7, characterized in that, After obtaining the multimedia resources to be checked for plagiarism, the process also includes: Extract a preset number of image frames to be checked from the multimedia resources to be checked for plagiarism. The step of performing image feature extraction on the image frames in the multimedia resource to be deduplicated, to obtain the image features of the image frames, includes: Image feature extraction is performed on the preset number of image frames to be deduplicated to obtain the image features of the image frames to be deduplicated.

9. A multimedia resource plagiarism detection device, characterized in that, include: The acquisition module is configured to acquire multimedia resources to be checked for plagiarism. The feature extraction module is configured to perform image feature extraction on image frames in the multimedia resource to be deduplicated, and obtain image features of the image frames; wherein, the image features include full-image features of the image frame and / or local image features, the local image features being the core motion region image features contained in the image frame, the core motion region being the region in the image frame containing the actual object being photographed; The search module is configured to search for full-image candidate features corresponding to the image features from a first feature library, and to search for local image candidate features corresponding to the image features from a second feature library; wherein, the first feature library stores full-image candidate features of image frames of multiple first candidate multimedia resources, and the second feature library stores local image candidate features of image frames of multiple second candidate multimedia resources. The determination module is configured to determine, based on the searched full-image candidate features and local image candidate features, the multimedia resources that duplicate the multimedia resource to be checked among the plurality of first candidate multimedia resources and the plurality of second candidate multimedia resources; The search module is configured as follows: From the first feature library, search for full-image candidate features whose feature similarity to the image features reaches a preset threshold, and from the second feature library, search for local image candidate features whose feature similarity to the image features reaches the preset threshold.

10. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the multimedia resource deduplication method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, When the instructions in the computer-readable storage medium are executed by the processor of the electronic device, the electronic device is able to perform the multimedia resource deduplication method as described in any one of claims 1 to 8.

Images