A method, apparatus, and computer device for fingerprint-based association identification of the main character.

Abstract

This application relates to a method, apparatus, and computer device for association recognition based on protagonist fingerprints. The method involves artificial intelligence and includes: acquiring video content data to be processed, including first video content data published by a first publishing object and second video content data published by a second publishing object; performing entity recognition based on the video content data to obtain corresponding entity recognition results; performing protagonist recognition based on the entity recognition results to determine the first protagonist fingerprint of the first publishing object and the second protagonist fingerprint of the second publishing object; and determining the degree of association between the first publishing object and the second publishing object based on the similarity between the first and second protagonist fingerprints. This method enables similarity recognition based on the more core protagonist fingerprint, reducing the storage of redundant information and the information loading time during data computation, thereby improving the accuracy of determining the degree of association between publishing objects on different content distribution platforms.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus and computer device for association recognition based on the protagonist's fingerprint. Background Technology

[0002] With the development of artificial intelligence technology and the widespread use of various video applications and content distribution platforms, more and more users are publishing and viewing videos through different software or platforms. As a result, the amount of video content is increasing day by day. In order to protect the original video content of platform users and maintain the stable operation of the platform, it is necessary to manage the massive amount of video content to avoid a large number of duplicate videos or infringement of the original video content of platform users.

[0003] Traditionally, deep learning models are often used to calculate the similarity of video content produced by different video self-media publishers (i.e., publishers of video content on different content distribution platforms) to identify similar video content and determine whether the two publishers of similar video content are the same or similar publishers, in order to determine whether there is an issue of copyright infringement of original video content.

[0004] However, the traditional method of judging similarity solely based on the video content itself still suffers from low accuracy in identifying identical or similar publishing objects due to the interference of various extraneous information in a large amount of video content. Summary of the Invention

[0005] Therefore, it is necessary to provide a method, apparatus, and computer device for identifying related objects based on protagonist fingerprints, which can improve the accuracy of the correlation between different published objects on different content distribution platforms.

[0006] Firstly, this application provides a method for identifying a subject based on fingerprint association. The method includes:

[0007] Acquire video content data to be processed, the video content data including first video content data published by a first publishing object and second video content data published by a second publishing object;

[0008] Entity recognition is performed based on the video content data to obtain the corresponding entity recognition result. The entity recognition result includes a first entity object corresponding to the first video content data and a second entity object corresponding to the second video content data.

[0009] Based on the entity recognition results, perform protagonist identification to determine the first protagonist fingerprint of the first publishing object and the second protagonist fingerprint of the second publishing object;

[0010] The degree of association between the first publishing object and the second publishing object is determined based on the similarity between the fingerprints of the first protagonist and the second protagonist.

[0011] Secondly, this application also provides a fingerprint-based association identification device. The device includes:

[0012] The video content data acquisition module is used to acquire video content data to be processed, the video content data including first video content data published by a first publishing object and second video content data published by a second publishing object;

[0013] An entity recognition module is used to perform entity recognition based on the video content data to obtain corresponding entity recognition results. The entity recognition results include a first entity object corresponding to the first video content data and a second entity object corresponding to the second video content data.

[0014] The protagonist identification module is used to identify the protagonist based on the entity identification result, and to determine the first protagonist fingerprint of the first publishing object and the second protagonist fingerprint of the second publishing object.

[0015] The association degree determination module is used to determine the association degree between the first publishing object and the second publishing object based on the similarity between the fingerprints of the first protagonist and the fingerprints of the second protagonist.

[0016] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0017] Acquire video content data to be processed, the video content data including first video content data published by a first publishing object and second video content data published by a second publishing object;

[0018] Entity recognition is performed based on the video content data to obtain the corresponding entity recognition result. The entity recognition result includes a first entity object corresponding to the first video content data and a second entity object corresponding to the second video content data.

[0019] Based on the entity recognition results, perform protagonist identification to determine the first protagonist fingerprint of the first publishing object and the second protagonist fingerprint of the second publishing object;

[0020] The degree of association between the first publishing object and the second publishing object is determined based on the similarity between the fingerprints of the first protagonist and the second protagonist.

[0021] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0022] Acquire video content data to be processed, the video content data including first video content data published by a first publishing object and second video content data published by a second publishing object;

[0023] Entity recognition is performed based on the video content data to obtain the corresponding entity recognition result. The entity recognition result includes a first entity object corresponding to the first video content data and a second entity object corresponding to the second video content data.

[0024] Based on the entity recognition results, perform protagonist identification to determine the first protagonist fingerprint of the first publishing object and the second protagonist fingerprint of the second publishing object;

[0025] The degree of association between the first publishing object and the second publishing object is determined based on the similarity between the fingerprints of the first protagonist and the second protagonist.

[0026] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0027] Acquire video content data to be processed, the video content data including first video content data published by a first publishing object and second video content data published by a second publishing object;

[0028] Entity recognition is performed based on the video content data to obtain the corresponding entity recognition result. The entity recognition result includes a first entity object corresponding to the first video content data and a second entity object corresponding to the second video content data.

[0029] Based on the entity recognition results, perform protagonist identification to determine the first protagonist fingerprint of the first publishing object and the second protagonist fingerprint of the second publishing object;

[0030] The degree of association between the first publishing object and the second publishing object is determined based on the similarity between the fingerprints of the first protagonist and the second protagonist.

[0031] In the aforementioned method, apparatus, computer device, and computer-readable storage medium for association recognition based on protagonist fingerprints, video content data to be processed is acquired, including first video content data published by a first publishing object and second video content data published by a second publishing object. Entity recognition is then performed based on the video content data to obtain corresponding entity recognition results. These entity recognition results include a first entity object corresponding to the first video content data and a second entity object corresponding to the second video content data. Protagonist recognition is then performed based on these entity recognition results to determine the first protagonist fingerprint of the first publishing object and the second protagonist fingerprint of the second publishing object. Furthermore, the degree of association between the first and second publishing objects can be determined based on the similarity between the first and second protagonist fingerprints. This enables further identification and mining of multiple entity objects in video content data in association recognition scenarios involving two publishing objects on different content distribution platforms. Interference from non-protagonist entity objects and other impurities is eliminated, and similarity recognition is performed based on the more core protagonist fingerprint. This reduces the storage of redundant information and the information loading time during data computation, while further improving the accuracy of the determined degree of association between publishing objects on different content distribution platforms. Attached Figure Description

[0032] Figure 1 This is an application environment diagram of a fingerprint-based association recognition method in one embodiment;

[0033] Figure 2 This is a flowchart illustrating a method for association and identification based on the protagonist's fingerprint in one embodiment;

[0034] Figure 3 This is a schematic diagram of the process for generating corresponding first object fingerprint data and second object fingerprint data in one embodiment;

[0035] Figure 4 This is a flowchart illustrating the association recognition method based on the protagonist's fingerprint in another embodiment;

[0036] Figure 5 This is a schematic diagram of the overall process of a fingerprint-based association identification method in one embodiment;

[0037] Figure 6 This is a structural block diagram of a fingerprint-based association recognition device in one embodiment;

[0038] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0040] The fingerprint-based association identification method provided in this application involves artificial intelligence (AI) technology. AI utilizes digital computers or computers-controlled machines to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to perceive, reason, and make decisions. AI technology is a comprehensive discipline involving a wide range of fields, encompassing both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0041] Machine Learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0042] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0043] The fingerprint-based association recognition method provided in this application specifically involves artificial intelligence technologies such as machine learning, and can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or located in the cloud or on other network servers. Server 104 acquires the video content data to be processed and performs entity recognition based on the video content data to obtain the corresponding entity recognition results. The video content data includes first video content data published by a first publishing object and second video content data published by a second publishing object. The entity recognition results include a first entity object corresponding to the first video content data and a second entity object corresponding to the second video content data. The video content data and entity recognition results can be stored locally on terminal 102 or in the data storage system corresponding to server 104. Furthermore, server 104 can perform protagonist identification based on the entity recognition results, determining the first protagonist fingerprint of the first publishing object and the second protagonist fingerprint of the second publishing object, and further determining the degree of association between the first and second publishing objects based on the similarity between the first and second protagonist fingerprints. After obtaining the degree of correlation between different publishing objects, similar or highly similar publishing objects can be further filtered to determine whether there are any copyright infringement issues related to original content. The corresponding degree of correlation and filtering information are then sent to terminal 102 for feedback or display. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers.

[0044] In one embodiment, such as Figure 2 As shown, a method for association and identification based on the protagonist's fingerprint is provided, which can be applied to... Figure 1 Taking the server in the example, the following steps are included:

[0045] Step S202: Obtain the video content data to be processed. The video content data includes the first video content data published by the first publishing object and the second video content data published by the second publishing object.

[0046] Specifically, based on content management requests from different content distribution platforms, similarity detection, content management, and correlation identification of different content publishers on these platforms are required. Specifically, for content publishers on different platforms, the correlation needs to be determined by combining the video content published by the publisher with the publisher's personal information, including name, profile picture, and bio.

[0047] Furthermore, the video content data includes first video content data published by a first publishing entity and second video content data published by a second publishing entity. Specifically, this requires data from two content distribution platforms: first video content data published by the first publishing entity on a first content distribution platform, and second video content data published by the second publishing entity on a second content distribution platform. It is understood that "first" and "second" are merely descriptive and not specifically limited. Also, the acquired first or second video content data can be one or more; that is, the first or second publishing entity can publish one or more video content data on their respective platforms.

[0048] Step S204: Perform entity recognition based on video content data to obtain the corresponding entity recognition results.

[0049] Specifically, the entity recognition result includes a first entity object corresponding to the first video content data and a second entity object corresponding to the second video content data. Specifically, it is necessary to extract the first keyframe corresponding to the first video content data and the second keyframe corresponding to the second video content data respectively, so as to perform entity object recognition based on the first keyframe and the second keyframe to obtain the first entity object corresponding to the first keyframe and the second entity object corresponding to the second keyframe.

[0050] Since it is necessary to identify and determine the degree of association between the first publishing object on the first content distribution platform and the second publishing object on the second content distribution platform, it is necessary to obtain video content data of different publishing objects on different content distribution platforms. That is, to perform similarity detection on the first video content data published by the first publishing object and the second video content data published by the second publishing object, so as to determine the degree of association between the two publishing objects based on the detected similarity, rather than obtaining video content data of different publishing objects on the same content distribution platform.

[0051] Furthermore, specifically, a keyframe extraction algorithm is used to extract keyframes based on the acquired video content data. Specifically, the algorithm extracts the first keyframe corresponding to the first video content data and the second keyframe corresponding to the second video content data. The keyframe extraction algorithm can be a shot-based extraction algorithm, a motion analysis-based extraction algorithm, or a video clustering-based extraction algorithm. This embodiment does not impose specific limitations; the requirement is met as long as keyframes can be successfully extracted from the video content data.

[0052] In one embodiment, an image object recognition algorithm can be used to perform entity object recognition processing on keyframes to obtain the entity objects carried by each keyword. Since the video content data includes multiple keyframes, the identified entity objects will differ for different keyframes. Therefore, further protagonist identification is needed for the identified entity objects to determine the main character of the video content data. The purpose of identifying the main character is to reduce interference factors such as non-main character entities in the video content data, thereby improving the accuracy of the final determined association degree. The image object recognition algorithm can be an R-CNN model, Fast-RCNN model, Faster-RCNN model, or YOLO model, etc. This embodiment does not limit the specific model; entity object recognition is achieved through these models, generating corresponding entity recognition results.

[0053] Specifically, an entity object refers to different human objects appearing in the video content data. This can include the publishing object that shoots and publishes the video content, as well as other human objects besides the publishing object that shoots and publishes the video content, such as other human objects that enter the current shooting area during the shooting process, or human objects that serve as background objects during the shooting process. Among these, all of these other human objects are non-main character entity objects.

[0054] Furthermore, since the extracted video content contains too much redundant information, such as multiple human or object entities appearing in a single frame of the video, when calculating the content similarity using the full information of that frame to obtain the similarity of the published objects, there is usually interference from extraneous information such as multiple non-main entity objects. Extraneous information in the video content data needs to be removed to improve the accuracy of the final determination of the degree of association between the published objects.

[0055] Step S206: Based on the entity recognition results, perform protagonist recognition to determine the first protagonist fingerprint of the first publishing object and the second protagonist fingerprint of the second publishing object.

[0056] Specifically, by performing content clustering processing on each identified first entity object and each second entity object, corresponding first object fingerprint data and second object fingerprint data are generated. Then, based on the protagonist recognition algorithm, the first protagonist fingerprint corresponding to the first publishing object can be determined from each first object fingerprint data, and the second protagonist fingerprint corresponding to the second publishing object can be determined from each second object fingerprint data.

[0057] Specifically, after determining the fingerprint of the first protagonist from the fingerprint data of the first object and the fingerprint of the second protagonist from the fingerprint data of the second object, a content-granular level first protagonist object fingerprint space and second protagonist object fingerprint space can be constructed based on the first protagonist fingerprint and the second protagonist fingerprint, respectively. The first protagonist object fingerprint space and the second protagonist object fingerprint space represent the fingerprint sets composed of protagonist fingerprints in the corresponding video content data.

[0058] Furthermore, entity recognition of the video content data can generate corresponding entity recognition results. These results can include entity objects identified from each keyframe of the video content data, specifically the first entity object corresponding to the first video content data and the second entity object corresponding to the second video content data. Since there are multiple entity objects in the video content data, a clustering algorithm is needed to cluster each entity object at the content granularity level to generate object fingerprint data. The clustering algorithm can be K-MEANS clustering, mean-shift clustering, hierarchical clustering, etc. This embodiment does not limit the specific clustering algorithm; any algorithm that meets the content clustering requirements is acceptable.

[0059] Granularity refers to the level of detail in data statistics within the same dimension. Content granularity can be understood as multiple entity objects within a single piece of content. By clustering these entity objects at the content granularity level, object fingerprint data can be generated to represent them. Object fingerprint data can be understood as a unique quantitative representation of various entity objects identified in video keyframes. An object fingerprint can originate from one or more similar and identical entity objects. The protagonist fingerprint, on the other hand, represents the protagonist entity object in the video content, typically represented by the publishing object of the video content data on a content distribution platform.

[0060] In one embodiment, a protagonist identification algorithm is used to determine a protagonist identification model that meets the identification requirements. Then, the protagonist identification model that meets the identification requirements is used to identify the protagonist in the object fingerprint data, so as to determine the corresponding protagonist fingerprint from each object fingerprint data.

[0061] Specifically, based on a protagonist identification model that meets the identification requirements, protagonist identification can be performed on the fingerprint data of each first object and each second object. The first protagonist fingerprint corresponding to the first publishing object can be determined from the fingerprint data of each first object, and the second protagonist fingerprint corresponding to the second publishing object can be determined from the fingerprint data of each second object. The protagonist identification model is used to identify the object fingerprint data of each entity object in the video content data to determine the protagonist fingerprint of the video's main character.

[0062] Furthermore, since the video protagonist refers to the main character in the video content, usually the creator of the video content data, the protagonist can be identified by an algorithm or manually identified and labeled by video content review staff. Therefore, an initial facial recognition model can be trained based on the protagonist's characteristics and facial features to obtain a trained protagonist recognition model. This trained model can then be used to identify the protagonist in the video content data.

[0063] In one embodiment, the step of training an initial human face recognition model to obtain a trained protagonist recognition model specifically includes:

[0064] Collect different facial images from the video content, count the frequency of each facial image, and obtain the pre-annotated facial features on each facial image; based on the frequency of each facial image and the pre-annotated facial features, determine the main character's facial image and the non-main character's facial images; obtain a training feature set based on the facial features of the main character's facial image and the non-main character's facial images; train the initial portrait recognition model based on the training feature set to obtain the trained main character recognition model.

[0065] Furthermore, the initial facial recognition model may include a recognition model based on geometric features, a face recognition model based on deep learning, a local feature analysis model, and an eigenface recognition model, etc. This embodiment does not impose specific limitations, and achieving facial recognition is sufficient to meet the requirements.

[0066] Among them, geometric feature-based face recognition methods require describing the relationships between facial features using geometric data, converting this data into feature vectors, and then performing face recognition. The recognition process specifically includes: recording facial geometric information based on the position and relative relationships of key features; extracting geometric features from the obtained information; and using an effective classifier to classify the features and identify the individual. Specifically, a face recognition model can be obtained by collecting geometric features from different faces as training features.

[0067] Similarly, deep learning-based face recognition models utilize convolutional neural networks (CNNs) to extract features, enabling the extraction and recognition of facial features, such as in specific recognition scenarios involving 3D face alignment and face representation.

[0068] Furthermore, Local Face Analysis models can include Local Number of Directions (LDN) recognition models, Local Binary Pattern (LBP) recognition models, and Local Graph Structure (LGS) recognition models, etc.

[0069] Among these, eigenface recognition models can include face recognition models based on principal component analysis (PCA). Specifically, PCA seeks to identify the fundamental elements of the distribution of face images from a statistical perspective—that is, the eigenvectors of the covariance matrix of the face image sample set. These eigenvectors are called eigenfaces, which approximate the face image. Eigenfaces reflect the information implicit within the face sample set and the structural relationships of the face. By collecting eigenfaces from different face images as training features for model training, a final recognition model for face recognition can be obtained.

[0070] Step S208: Determine the degree of association between the first publishing object and the second publishing object based on the similarity between the fingerprints of the first protagonist and the second protagonist.

[0071] Specifically, the similarity between the fingerprints of the first and second main characters is calculated, and based on this similarity, the degree of association between the first and second publishing objects is calculated to determine their correlation. More specifically, the degree of association between the first and second publishing objects is calculated based on the difference between the similarity score and a preset similarity threshold. That is, the degree of association between the first and second publishing objects is determined by calculating the value by which the similarity score exceeds the preset similarity threshold. Furthermore, the degree of association can be ranked from highest to lowest based on the magnitude of the association value. The larger the value by which the similarity score exceeds the preset similarity threshold, the higher the degree of association between the first and second publishing objects.

[0072] Specifically, the degree of correlation between the first and second publishing objects determines whether they are the same or similar. If the correlation indicates that the publishing objects on the two content distribution platforms are the same, it means that the same publishing object is publishing video content on different content distribution platforms. However, if the video content is similar or identical, but the first and second publishing objects are not the same, it indicates an infringement of original video content. This needs to be reported to the platform for content management and publishing object management. Similarly, relevant prompts can be sent to the corresponding publishing objects to remind users to take subsequent actions such as deleting the infringing content.

[0073] In one embodiment, a preset similarity threshold can be obtained, and the preset similarity threshold and the calculated similarity can be compared to determine whether the calculated similarity is greater than the preset similarity threshold.

[0074] Specifically, when the calculated similarity is greater than a preset similarity threshold, the association between the first and second publishing objects is determined to be strong. Conversely, when the calculated similarity is less than the preset similarity threshold, the association between the first and second publishing objects is determined to be weak, meaning that in this case, the probability that the first and second publishing objects belong to the same publishing object is relatively low.

[0075] Furthermore, when it is determined that the calculated similarity is greater than the preset similarity threshold, the value of the similarity exceeding the preset similarity threshold is further calculated, and the degree of association is ranked according to the value of the similarity exceeding the preset similarity threshold. That is, the larger the value of the similarity exceeding the preset similarity threshold, the higher the degree of association between the first publishing object and the second publishing object.

[0076] In one embodiment, determining the similarity between the fingerprints of the first protagonist and the fingerprints of the second protagonist includes using a similarity algorithm to calculate the similarity between the fingerprints of the first protagonist and the fingerprints of the second protagonist.

[0077] The similarity algorithm can be calculated by using methods such as Pearson correlation coefficient, Euclidean distance, Cosine similarity, Manhattan distance, and log-likelihood similarity between the fingerprints of the first and second protagonists. In this embodiment, no specific limitation is made on the similarity algorithm, as long as it meets the requirements for similarity calculation.

[0078] Furthermore, the keyframes included in the first and second video content data are stored in association through a first mapping table. That is, the first video content data and the first keyframe, as well as the second video content data and the second keyframe, are all stored in association through the first mapping table. Then, based on the identified first entity objects in each first keyframe, the second entity objects in each second keyframe, and the first mapping table, a second mapping table can be established between the first video content data and each first entity object, and between each second video content data and each second entity object.

[0079] Specifically, based on the second mapping table, clustering is performed on each entity object at the content granularity level to generate corresponding object fingerprint data, and a third mapping table is further established between video content data and object fingerprint data. That is, based on the second mapping table, clustering is performed on each first entity object and each second entity object at the content granularity level to generate corresponding first and second object fingerprint data.

[0080] In one embodiment, based on the protagonist recognition algorithm, a protagonist recognition model that meets the recognition requirements can be determined. Then, based on the protagonist recognition model that meets the recognition requirements, protagonist recognition can be performed on each first object fingerprint data and each second object fingerprint data to determine the first protagonist fingerprint corresponding to the first publishing object from each first object fingerprint data, and the second protagonist fingerprint corresponding to the second publishing object from each second object fingerprint data.

[0081] Then, based on the third mapping table, as well as the fingerprints of the first and second protagonists, a fourth mapping table is established between the first video content data and the fingerprints of the first and second protagonists, and the second video content data and the fingerprints of the second protagonists.

[0082] The first video content data and the first publishing object, as well as the second video content data and the second publishing object, are stored together through the fifth mapping relationship list. Based on the fifth mapping relationship table and the fourth mapping relationship table, the first publishing object and the first protagonist fingerprint, the second publishing object and the second protagonist fingerprint are fused together to generate the sixth mapping relationship table between the first publishing object and the first protagonist fingerprint, the second publishing object and the second protagonist fingerprint.

[0083] Furthermore, according to the sixth mapping table, the first object number of the first publishing object and the second object number of the second publishing object can be determined as text numbers, and the first protagonist fingerprint and the second protagonist fingerprint can be determined as text data. Then, based on the text numbers and text data, an index relationship is established between the first publishing object and the first protagonist fingerprint, and between the second publishing object and the second protagonist fingerprint. By establishing the index relationship, the corresponding first protagonist fingerprint or second protagonist fingerprint can be obtained for any first publishing object or any second publishing object, and the similarity between the first protagonist fingerprint and the second protagonist fingerprint can be calculated.

[0084] In one embodiment, after determining the degree of association between the first publishing object and the second publishing object based on the similarity between the fingerprints of the first protagonist and the second protagonist, the method further includes:

[0085] Sort the objects according to their degree of association and generate a corresponding association degree sequence; determine the maximum degree of association based on the association degree sequence, and obtain the first and second publishing objects corresponding to the maximum degree of association, and determine them as the same publishing object.

[0086] Specifically, the objects are sorted according to their degree of association, and a corresponding association sequence is generated. For example, the first and second publishing objects with the highest degree of association are at the beginning of the association sequence, while the first and second publishing objects with the lowest degree of association are at the end of the association sequence.

[0087] Furthermore, the maximum degree of association can be determined based on the association degree sequence. For example, the one at the beginning of the association degree sequence is the maximum degree of association. Then, the first and second publishing objects corresponding to the maximum degree of association can be obtained and determined to be the same publishing object.

[0088] In one embodiment, after determining the degree of association between the first publishing object and the second publishing object based on the similarity between the fingerprints of the first protagonist and the second protagonist, the method further includes:

[0089] Based on the preset correlation threshold and each correlation level, the correlation level that meets the judgment requirements is determined; the first publishing object and the second publishing object corresponding to the correlation level that meets the judgment requirements are determined as the same publishing object.

[0090] Specifically, by obtaining a preset correlation threshold and comparing each correlation level with the preset correlation threshold, correlation levels greater than the preset correlation threshold are determined, and these correlation levels are identified as those meeting the judgment requirements. Furthermore, the first and second publishing objects corresponding to the correlation levels meeting the judgment requirements—that is, the first and second publishing objects corresponding to the correlation levels greater than the preset correlation threshold—are identified as the same publishing object.

[0091] The aforementioned association recognition method based on protagonist fingerprints acquires video content data to be processed, including first video content data published by a first publishing object and second video content data published by a second publishing object. Entity recognition is then performed based on the video content data to obtain corresponding entity recognition results. These entity recognition results include a first entity object corresponding to the first video content data and a second entity object corresponding to the second video content data. Protagonist identification is then performed based on these entity recognition results to determine the first protagonist fingerprint of the first publishing object and the second protagonist fingerprint of the second publishing object. Furthermore, the degree of association between the first and second publishing objects can be determined based on the similarity between the first and second protagonist fingerprints. This method enables further identification and mining of multiple entity objects in video content data in association recognition scenarios involving two publishing objects on different content distribution platforms. It eliminates interference from non-protagonist entity objects and other impurities, and performs similarity recognition based on the core protagonist fingerprint. This reduces the storage of redundant information and the information loading time during data computation, while further improving the accuracy of the determined degree of association between publishing objects on different content distribution platforms.

[0092] In one embodiment, such as Figure 3 As shown, the steps for generating corresponding first object fingerprint data and second object fingerprint data, namely, performing content clustering processing on each identified first entity object and each second entity object to generate corresponding first object fingerprint data and second object fingerprint data, specifically include:

[0093] Step S302: Obtain the first mapping relationship table between the first video content data and the first keyframe, and between the second video content data and the second keyframe.

[0094] Specifically, after acquiring the first video content data published by the first publishing object and the second video content data published by the second publishing object, and extracting keyframes from the first video content data and the second video content, a first mapping relationship table between the first video content data and the first keyframe, the second video content and the second keyframe is established. Then, when it is necessary to perform content clustering processing based on entity recognition results, the first mapping relationship table between the first video content data and the first keyframe, the second video content data and the second keyframe can be directly obtained.

[0095] The first mapping table includes the first video content data number, the first keyframe, the second video content data number, the second keyframe, and the mapping relationships between the first video content data number and the first keyframe, and between the second video content data number and the second keyframe.

[0096] Step S304: Based on the first entity objects in each first keyframe, the second entity objects in each second keyframe, and the first mapping table, establish a second mapping table between the first video content data and each first entity object, and between each second video content data and each second entity object.

[0097] Specifically, based on the first entity objects identified from each first keyframe, the second entity objects identified from each second keyframe, and the first mapping relationship table between the first video content data and the first keyframe, and between the second video content data and the second keyframe, a second mapping relationship table is further established between the first video content data and the first entity objects, and between the second video content data and the second entity objects.

[0098] The purpose of establishing the first mapping table is to prepare for extracting the first entity object and the second entity object from the first keyframe and the second keyframe. It can be understood that, through the first mapping table and the keyframe extraction algorithm, a second mapping table can be established between the first video content data and each first entity object extracted from the first keyframe, and between the second video content data and each second entity object extracted from the second keyframe. The second mapping table includes the mapping relationships between the first video content data number and the first entity object, the second video content data number and the second entity object, and the mapping relationships between the first video content data number and the first entity object, and the second video content data number and the second entity object. Its function is to prepare for the subsequent establishment of a third mapping table between the first video content data and the first object fingerprint data, and between the second video content data and the second object fingerprint data.

[0099] Step S306: Based on the second mapping relationship table, clustering is performed on each first entity object and each second entity object at the content granularity level to generate corresponding first object fingerprint data and second object fingerprint data, and a third mapping relationship table is established between the first video content data and the first object fingerprint data, the second video content data and the second object fingerprint data.

[0100] Specifically, based on the second mapping relationship table between the first video content data and each first entity object, and between each second video content data and each second entity object, clustering is performed on each first entity object and each second entity object at the content granularity level to generate corresponding first object fingerprint data and second object fingerprint data. Here, object fingerprint data can be understood as a unique quantitative representation of various entity objects identified in video keyframes, and an object fingerprint can originate from one or more similar and identical entity objects.

[0101] Furthermore, after generating the first object fingerprint data and the second object fingerprint data, a third mapping relationship table is established based on the second mapping relationship table between the first video content data and each first entity object, and between each second video content data and each second entity object, as well as the first object fingerprint data and the second object fingerprint data.

[0102] The third mapping table contains the first video content data number and the first object fingerprint data, the second video content data number and the second object fingerprint data, as well as the mapping relationship between the first video content data number and the first object fingerprint data, and between the second video content data number and the second object fingerprint data. Its function is to prepare for the further generation of the fourth mapping table between the first video content data and the first protagonist fingerprint, the second video content data and the second protagonist fingerprint.

[0103] In this embodiment, a first mapping table is obtained between the first video content data and the first keyframe, and between the second video content data and the second keyframe. Based on the identified first entity objects in each first keyframe, second entity objects in each second keyframe, and the first mapping table, a second mapping table is established between the first video content data and each first entity object, and between each second video content data and each second entity object. Then, based on the second mapping table, clustering is performed on each first entity object and each second entity object at the content granularity level to generate corresponding first object fingerprint data and second object fingerprint data. A third mapping table is then established between the first video content data and the first object fingerprint data, and between the second video content data and the second object fingerprint data. This achieves entity recognition and main character recognition of the video content data, eliminating interference from non-main character entity objects and other impurity information. This allows for subsequent similarity recognition based on the more core main character fingerprint, reducing redundant information storage and loading time, thereby improving the accuracy and processing efficiency of similarity recognition.

[0104] In one embodiment, the steps of determining the first protagonist fingerprint corresponding to the first publishing object from the fingerprint data of each first object, and determining the second protagonist fingerprint corresponding to the second publishing object from the fingerprint data of each second object, based on the protagonist recognition algorithm, include:

[0105] Based on the protagonist recognition algorithm, a protagonist recognition model that meets the recognition requirements is determined;

[0106] Based on the protagonist recognition model that meets the recognition requirements, protagonist recognition is performed on the fingerprint data of each first object and the fingerprint data of each second object. The first protagonist fingerprint corresponding to the first publishing object is determined from the fingerprint data of each first object, and the second protagonist fingerprint corresponding to the second publishing object is determined from the fingerprint data of each second object. The protagonist recognition model that meets the recognition requirements is obtained by training the initial human face recognition model.

[0107] Based on the third mapping table, as well as the fingerprints of the first and second protagonists, a fourth mapping table is established between the first video content data and the fingerprints of the first and second protagonists, and between the second video content data and the fingerprints of the second protagonist.

[0108] Specifically, based on the protagonist recognition algorithm, a protagonist recognition model that meets the recognition requirements is determined. The protagonist recognition algorithm is used to determine a protagonist recognition model that can perform protagonist recognition on video content data. Meeting the recognition requirements means that the protagonist recognized by the protagonist recognition model must be the creator of the video content data, rather than other non-protagonist entities appearing in the video content data. That is, other human objects other than the publishing object that shoots and publishes the video content, such as other human objects that enter the current shooting area during the shooting process, or human objects that are background objects during the shooting process.

[0109] In one embodiment, different facial images are collected from video content, the frequency of each facial image is counted, and pre-annotated facial features are obtained on each facial image. Then, based on the frequency of each facial image and the pre-annotated facial features, the main character's facial image and non-main character's facial images are determined. A training feature set is obtained based on the facial features of the main character's facial image and non-main character's facial images. The initial portrait recognition model is trained using the training feature set to obtain a trained main character recognition model. Then, based on the trained main character recognition model, such as a trained deep learning-based main character recognition model, main character recognition is performed on each first object fingerprint data and second object fingerprint data. The corresponding first main character fingerprint is determined from each first object fingerprint data, and the corresponding second main character fingerprint is determined from each second object fingerprint data.

[0110] Furthermore, by obtaining a third mapping relationship table between the first video content data and the first object fingerprint data, the second video content data and the second object fingerprint data, and based on the third mapping relationship table, the first protagonist fingerprint and the second protagonist fingerprint, a fourth mapping relationship table is established between the first video content data and the first protagonist fingerprint, the second video content data and the second protagonist fingerprint.

[0111] The fourth mapping table includes the first video content data number and the first protagonist fingerprint, the second video content data number and the second protagonist fingerprint, as well as the correspondence between the first video content data number and the first protagonist fingerprint, and the second video content data number and the second protagonist fingerprint. Its function is to prepare for the subsequent generation of the sixth mapping table between the first publishing object and the first protagonist fingerprint, and the second publishing object and the second protagonist fingerprint by combining the fifth mapping table between the first video content data and the first publishing object, and the fifth mapping table between the second video content data and the second publishing object.

[0112] In this embodiment, based on the protagonist recognition algorithm, a protagonist recognition model that meets the recognition requirements is determined. Then, based on this model, protagonist recognition is performed on the fingerprint data of each first object and each second object. The first protagonist fingerprint corresponding to the first publishing object is determined from the fingerprint data of each first object, and the second protagonist fingerprint corresponding to the second publishing object is determined from the fingerprint data of each second object. Furthermore, based on the third mapping table, the first protagonist fingerprint, and the second protagonist fingerprint, a fourth mapping table is established between the first video content data and the first protagonist fingerprint, and between the second video content data and the second protagonist fingerprint. This achieves protagonist recognition of the object fingerprint data, identifying the protagonist fingerprint and eliminating interference from non-protagonist entities, non-protagonist fingerprint data, and other impurities, so that subsequent similarity recognition can be performed based on the more core protagonist fingerprint, improving the accuracy of similarity recognition.

[0113] In one embodiment, after establishing a fourth mapping table between the first video content data and the first protagonist fingerprint, the second video content data, and the second protagonist fingerprint based on the third mapping table, the first protagonist fingerprint, and the second protagonist fingerprint, the method further includes:

[0114] Obtain the first video content data and the first publishing object, as well as the fifth mapping table between the second video content data and the second publishing object;

[0115] Based on the fourth and fifth mapping relationship tables, data fusion processing is performed on the first publishing object and the first protagonist fingerprint, the second publishing object and the second protagonist fingerprint to generate a sixth mapping relationship table between the first publishing object and the first protagonist fingerprint, the second publishing object and the second protagonist fingerprint.

[0116] Specifically, a fifth mapping table exists between the first video content data and the first publishing object, as well as between the second video content data and the second publishing object. That is, the first video content data and the first publishing object, as well as the second video content data and the second publishing object, are stored together through this fifth mapping table. By obtaining the fifth mapping table between the first video content data and the first publishing object, as well as between the second video content data and the second publishing object, and combining the fourth mapping table between the first video content data and the first protagonist fingerprint, and the second video content data and the second protagonist fingerprint, with the fifth mapping table, data fusion processing is performed on the first publishing object and the first protagonist fingerprint, and the second publishing object and the second protagonist fingerprint, to generate a sixth mapping table between the first publishing object and the first protagonist fingerprint, and the second publishing object and the second protagonist fingerprint.

[0117] Furthermore, the sixth mapping table includes the first publishing object and the first protagonist fingerprint, the second publishing object and the second protagonist fingerprint, as well as the mapping relationships between the first publishing object and the first protagonist fingerprint, and between the second publishing object and the second protagonist fingerprint. Its function is that after obtaining the object numbers of the first publishing object and the second publishing object, the first protagonist fingerprint corresponding to the first publishing object and the second protagonist fingerprint corresponding to the second publishing object can be obtained from the table according to the sixth mapping table, and the similarity between the first protagonist fingerprint and the second protagonist fingerprint can be calculated, so as to further determine the degree of association between the first publishing object and the second publishing object based on the similarity between the first protagonist fingerprint and the second protagonist fingerprint.

[0118] In this embodiment, by acquiring the first video content data and the first publishing object, as well as the fifth mapping table between the second video content data and the second publishing object, data fusion processing is performed on the first publishing object and the first protagonist fingerprint, and the second publishing object and the second protagonist fingerprint, based on the fourth and fifth mapping tables, to generate a sixth mapping table between the first publishing object and the first protagonist fingerprint, and the second publishing object and the second protagonist fingerprint. This achieves joint consideration of the mapping relationship between the publishing object itself and the video content data, and the mapping relationship between the video content data and the protagonist fingerprint, avoiding the situation of only considering simple information about the publishing object itself or a single video content data. This provides more comprehensive data information and mapping relationships for the subsequent protagonist fingerprint similarity calculation process, improving the accuracy of the calculated similarity.

[0119] In one embodiment, such as Figure 4 As shown, a method for association recognition based on protagonist fingerprints is provided. Before determining the degree of association between the first publishing object and the second publishing object based on the similarity between the fingerprints of the first protagonist and the second protagonist, the method further includes:

[0120] Step S402: According to the sixth mapping relationship table, the first object number of the first publishing object and the second object number of the second publishing object are determined as text numbers, and the first protagonist fingerprint and the second protagonist fingerprint are determined as text data.

[0121] Specifically, based on the fifth mapping relationship table between the first video content data and the first publishing object, the second video content data and the second publishing object, and the fourth mapping relationship table between the first video content data and the first protagonist fingerprint, the second video content data and the second protagonist fingerprint, data fusion processing can be performed on the first publishing object and the first protagonist fingerprint, the second publishing object and the second protagonist fingerprint to generate a sixth mapping relationship table between the first publishing object and the first protagonist fingerprint, the second publishing object and the second protagonist fingerprint.

[0122] Furthermore, the first publishing object and the first protagonist fingerprint, as well as the second publishing object and the second protagonist fingerprint, included in the sixth mapping relationship table can be obtained. The first object number of the first publishing object and the second object number of the second publishing object are determined as text numbers, and the first protagonist fingerprint and the second protagonist fingerprint are determined as text data.

[0123] Step S404: Based on the text number and text data, establish the index relationship between the first publishing object and the first protagonist fingerprint, the second publishing object and the second protagonist fingerprint.

[0124] In this approach, an index relationship can be established between text numbers and text data. This means that the corresponding text data can be retrieved based on the text number without sequentially searching for text data in the database, thus improving the efficiency of text data retrieval. Furthermore, an index relationship can be established between the first publishing object and its first and second publishing objects, as well as the first and second main character fingerprints, which are determined to be text numbers.

[0125] Step S406: Based on the established index relationship, obtain the first protagonist fingerprint corresponding to the first publishing object and the second protagonist fingerprint corresponding to the second publishing object.

[0126] Specifically, based on the established index relationship, according to the first publishing object, the fingerprint set formed by the fingerprint of the first protagonist in the first video content data is obtained from the fingerprint space of the first protagonist object, that is, the fingerprint of the first protagonist corresponding to the first publishing object is obtained; and according to the second publishing object, the fingerprint set formed by the fingerprint of the second protagonist in the second video content data is obtained from the fingerprint space of the second protagonist object, that is, the fingerprint of the second protagonist corresponding to the second publishing object is obtained.

[0127] Step S408: Calculate the similarity between the obtained first protagonist fingerprint and second protagonist fingerprint.

[0128] Specifically, the method for determining the similarity between the fingerprints of the first and second main characters includes calculating the similarity between the fingerprints of the first and second main characters using a similarity algorithm. The similarity algorithm can be calculated using methods such as Pearson correlation coefficient, Euclidean distance, Cosine similarity, Manhattan distance, and log-likelihood similarity. This embodiment does not specify a particular similarity algorithm, as long as it meets the requirements for similarity calculation.

[0129] In one embodiment, after calculating the similarity between the acquired first protagonist fingerprint and second protagonist fingerprint, the method further includes:

[0130] Based on the similarity between the fingerprints of the first protagonist and the second protagonist, the degree of association between the first publishing object and the second publishing object is calculated to determine the degree of association between the first publishing object and the second publishing object.

[0131] Furthermore, based on the degree of correlation between the first and second publishing objects, it can be determined whether the first and second publishing objects are the same or similar. When the correlation between the first and second publishing objects determines that the publishing objects on the two content distribution platforms are the same, it indicates that the same publishing object is publishing video content on different content distribution platforms. However, if the video content is similar or the same, but the first and second publishing objects are not the same, it indicates that there is an issue of copyright infringement of the original video content. This needs to be reported to the platform for content management and publishing object management. Similarly, corresponding prompts can be sent to the corresponding publishing objects to remind users to perform subsequent processing operations such as deleting the infringing content.

[0132] In this embodiment, based on the sixth mapping relationship table, the first object number of the first publishing object and the second object number of the second publishing object are determined as text numbers, and the first protagonist fingerprint and the second protagonist fingerprint are determined as text data. Then, based on the text numbers and text data, an index relationship is established between the first publishing object and the first protagonist fingerprint, and between the second publishing object and the second protagonist fingerprint. Finally, based on the established index relationship, the first protagonist fingerprint corresponding to the first publishing object and the second protagonist fingerprint corresponding to the second publishing object are obtained, so as to further calculate the similarity between the obtained first protagonist fingerprint and second protagonist fingerprint. This achieves the calculation of the similarity between the first protagonist fingerprint and the second protagonist fingerprint corresponding to the first publishing object and the second publishing object, rather than simply calculating the similarity between video content. Therefore, when determining the degree of association between publishing objects, a more accurate similarity judgment can be made, improving the accuracy of the determined degree of association between publishing objects.

[0133] In one embodiment, such as Figure 5 As shown, an overall process for a fingerprint-based association recognition method is provided, referring to... Figure 5 It can be seen that the method specifically includes the following steps:

[0134] Step S501: Obtain the video content data to be processed, including the first video content data published by the first publishing object and the second video content data published by the second publishing object, and extract the first keyframe corresponding to the first video content data and the second keyframe corresponding to the second video content data respectively.

[0135] Step S502: Entity object recognition is performed based on the first keyframe and the second keyframe to obtain the first entity object corresponding to the first keyframe and the second entity object corresponding to the second keyframe.

[0136] Step S503: Obtain the first mapping relationship table between the first video content data and the first keyframe, and between the second video content data and the second keyframe.

[0137] Step S504: Based on the first entity objects in each first keyframe, the second entity objects in each second keyframe, and the first mapping table, establish a second mapping table between the first video content data and each first entity object, and between each second video content data and each second entity object.

[0138] Step S505: Based on the second mapping relationship table, clustering is performed on each first entity object and each second entity object at the content granularity level to generate corresponding first object fingerprint data and second object fingerprint data, and a third mapping relationship table is established between the first video content data and the first object fingerprint data, the second video content data and the second object fingerprint data.

[0139] Step S506: Based on the protagonist recognition algorithm, determine the protagonist recognition model that meets the recognition requirements, and perform protagonist recognition on the fingerprint data of each first object and the fingerprint data of each second object according to the protagonist recognition model that meets the recognition requirements. Determine the first protagonist fingerprint corresponding to the first publishing object from the fingerprint data of each first object, and determine the second protagonist fingerprint corresponding to the second publishing object from the fingerprint data of each second object.

[0140] Step S507: Based on the third mapping relationship table, the first protagonist fingerprint, and the second protagonist fingerprint, establish a fourth mapping relationship table between the first video content data and the first protagonist fingerprint, the second video content data, and the second protagonist fingerprint.

[0141] Step S508: Obtain the first video content data and the first publishing object, as well as the fifth mapping relationship table between the second video content data and the second publishing object.

[0142] Step S509: Based on the fourth and fifth mapping relationship tables, perform data fusion processing on the first publishing object and the first protagonist fingerprint, the second publishing object and the second protagonist fingerprint, to generate a sixth mapping relationship table between the first publishing object and the first protagonist fingerprint, the second publishing object and the second protagonist fingerprint.

[0143] Step S510: According to the sixth mapping relationship table, the first object number of the first publishing object and the second object number of the second publishing object are determined as text numbers, and the first protagonist fingerprint and the second protagonist fingerprint are determined as text data.

[0144] Step S511: Based on the text number and text data, establish the index relationship between the first publishing object and the first protagonist fingerprint, the second publishing object and the second protagonist fingerprint.

[0145] Step S512: Based on the established index relationship, obtain the first protagonist fingerprint corresponding to the first publishing object and the second protagonist fingerprint corresponding to the second publishing object.

[0146] Step S513: Calculate the similarity between the obtained first protagonist fingerprint and second protagonist fingerprint, and determine the degree of association between the first publishing object and the second publishing object based on the similarity between the first protagonist fingerprint and the second protagonist fingerprint.

[0147] After executing step S513, execute steps S514 to S515. Step S514: Sort according to the degree of association and generate the corresponding degree of association sequence.

[0148] Step S515: Determine the maximum degree of association based on the degree of association sequence, and obtain the first publishing object and the second publishing object corresponding to the maximum degree of association, and determine them as the same publishing object.

[0149] After executing step S513, execute step S516. Step S516: Based on the preset correlation degree threshold and each correlation degree, determine the correlation degree that meets the judgment requirements, and determine the first publishing object and the second publishing object corresponding to the correlation degree that meets the judgment requirements as the same publishing object.

[0150] The aforementioned association recognition method based on protagonist fingerprints involves acquiring video content data to be processed, including first video content data published by a first publishing object and second video content data published by a second publishing object. Entity recognition is then performed based on the video content data to obtain corresponding entity recognition results. These entity recognition results include a first entity object corresponding to the first video content data and a second entity object corresponding to the second video content data. Protagonist recognition is then performed based on these entity recognition results to determine the first protagonist fingerprint of the first publishing object and the second protagonist fingerprint of the second publishing object. Furthermore, the degree of association between the first and second publishing objects can be determined based on the similarity between their fingerprints. This method enables further identification and mining of multiple entity objects in video content data in association recognition scenarios involving two publishing objects on different content distribution platforms. It eliminates interference from non-protagonist entity objects and other impurities, and performs similarity recognition based on the core protagonist fingerprint. This reduces the storage of redundant information and the information loading time during data computation, while further improving the accuracy of determining the degree of association between publishing objects on different content distribution platforms.

[0151] In one embodiment, this application also provides an application scenario for original content recognition, which utilizes the aforementioned protagonist fingerprint-based association recognition method. Specifically, the protagonist fingerprint-based association recognition method is applied in this scenario as follows:

[0152] Supporting original content published by various publishers on content distribution platforms is essential for building a vibrant and meaningful content ecosystem, and identifying original video publishers is a necessary prerequisite. The aforementioned fingerprint-based association identification method can identify similar or identical publishers across the entire network, thereby identifying potential pseudo-original video publishers and ensuring that resources are truly allocated to original video publishers, thus contributing to the healthy development of the content ecosystem on content distribution platforms.

[0153] In one embodiment, this application also provides an application scenario, which can be a copyright management scenario, in which the above-described fingerprint-based association identification method is applied. Specifically, the fingerprint-based association identification method is applied in this scenario as follows:

[0154] Copyright protection and management of video content data published by various publishers on content distribution platforms is a fundamental capability that content distribution platforms need to provide. Among them, the aforementioned association identification method based on protagonist fingerprints can identify the same or similar publishers across platforms and the entire network based on the protagonist fingerprints in the video content data. This helps content platforms manage copyrighted content, pay attention to potential pirated content, and safeguard the legitimate rights and interests of platform users.

[0155] In terms of business operations, original content identification and copyright management are two distinct standards. Original content identification focuses more on the identical content generated by various publishers on the content distribution platform, while copyright management emphasizes exclusivity. Specifically, the application of identical account identification in original content management is to monitor the corresponding off-site behavior of active original content creators on the platform, which can be used to assist in strategies, such as determining exclusivity. Copyright management, on the other hand, is used to identify identical content and theft, but original content management does not focus on identifying theft.

[0156] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0157] Based on the same inventive concept, this application also provides a protagonist fingerprint-based association identification device for implementing the aforementioned protagonist fingerprint-based association identification method. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations of one or more protagonist fingerprint-based association identification device embodiments provided below can be found in the limitations of the protagonist fingerprint-based association identification method described above, and will not be repeated here.

[0158] In one embodiment, such as Figure 6 As shown, a device for association recognition based on the protagonist's fingerprint is provided, including: a video content data acquisition module 602, an entity recognition module 604, a protagonist recognition module 606, and an association degree determination module 608, wherein:

[0159] The video content data acquisition module 602 is used to acquire video content data to be processed. The video content data includes first video content data published by a first publishing object and second video content data published by a second publishing object.

[0160] The entity recognition module 604 is used to perform entity recognition based on video content data to obtain the corresponding entity recognition results. The entity recognition results include a first entity object corresponding to the first video content data and a second entity object corresponding to the second video content data.

[0161] The protagonist identification module 606 is used to identify the protagonist based on the entity identification result and determine the first protagonist fingerprint of the first publishing object and the second protagonist fingerprint of the second publishing object.

[0162] The association degree determination module 608 is used to determine the association degree between the first publishing object and the second publishing object based on the similarity between the fingerprints of the first protagonist and the fingerprints of the second protagonist.

[0163] In the aforementioned fingerprint-based association identification device,

[0164] By acquiring video content data to be processed, including first video content data published by a first publishing object and second video content data published by a second publishing object, and performing entity recognition based on the video content data, corresponding entity recognition results are obtained. The entity recognition results include the first entity object corresponding to the first video content data and the second entity object corresponding to the second video content data. Then, based on the entity recognition results, protagonist identification is performed to determine the first protagonist fingerprint of the first publishing object and the second protagonist fingerprint of the second publishing object. Furthermore, the degree of association between the first and second publishing objects can be determined based on the similarity between the first and second protagonist fingerprints. This enables further identification and mining of multiple entity objects in video content data in scenarios involving the association identification of two publishing objects on different content distribution platforms. It removes interference from non-protagonist entity objects and other impurities, and performs similarity identification based on the more core protagonist fingerprint. This reduces the storage of redundant information and the information loading time during data computation, while further improving the accuracy of determining the degree of association between publishing objects on different content distribution platforms.

[0165] In one embodiment, the entity recognition module includes:

[0166] The keyframe extraction unit is used to extract the first keyframe corresponding to the first video content data and the second keyframe corresponding to the second video content data, respectively.

[0167] The entity object generation module is used to identify entity objects based on the first keyframe and the second keyframe, and to obtain the first entity object corresponding to the first keyframe and the second entity object corresponding to the second keyframe.

[0168] In one embodiment, the protagonist fingerprint determination module includes:

[0169] The object fingerprint data generation unit is used to perform content clustering processing on each identified first entity object and each second entity object to generate corresponding first object fingerprint data and second object fingerprint data.

[0170] The protagonist fingerprint determination unit is used to determine the first protagonist fingerprint corresponding to the first publishing object from the fingerprint data of each first object based on the protagonist recognition algorithm, and to determine the second protagonist fingerprint corresponding to the second publishing object from the fingerprint data of each second object.

[0171] In one embodiment, the object fingerprint data generation unit is further configured to:

[0172] Obtain the first video content data and the first keyframe, as well as the first mapping table between the second video content data and the second keyframe;

[0173] Based on the first entity objects in each first keyframe, the second entity objects in each second keyframe, and the first mapping table, establish a second mapping table between the first video content data and each first entity object, and between each second video content data and each second entity object.

[0174] Based on the second mapping table, clustering is performed on each first entity object and each second entity object at the content granularity level to generate corresponding first object fingerprint data and second object fingerprint data, and a third mapping table is established between the first video content data and the first object fingerprint data, the second video content data and the second object fingerprint data.

[0175] In one embodiment, the protagonist fingerprint determination unit is further configured to:

[0176] The fingerprint data of the second object is used for protagonist identification. The first protagonist fingerprint corresponding to the first publishing object is determined from the fingerprint data of each first object, and the second protagonist fingerprint corresponding to the second publishing object is determined from the fingerprint data of each second object. The protagonist identification model that meets the identification requirements is obtained by training the initial human face recognition model.

[0177] Based on the third mapping table, as well as the fingerprints of the first and second protagonists, a fourth mapping table is established between the first video content data and the fingerprints of the first and second protagonists, and between the second video content data and the fingerprints of the second protagonist.

[0178] In one embodiment, a fingerprint-based association identification device is provided, further comprising:

[0179] The fifth mapping relationship table acquisition module is used to acquire the first video content data and the first publishing object, as well as the fifth mapping relationship table between the second video content data and the second publishing object;

[0180] The sixth mapping relationship table generation module is used to perform data fusion processing on the first publishing object and the first protagonist fingerprint, the second publishing object and the second protagonist fingerprint based on the fourth mapping relationship table and the fifth mapping relationship table, and generate the sixth mapping relationship table between the first publishing object and the first protagonist fingerprint, the second publishing object and the second protagonist fingerprint.

[0181] In one embodiment, a fingerprint-based association identification device is provided, further comprising:

[0182] The text number and text data determination module is used to determine the first object number of the first publishing object and the second object number of the second publishing object as text numbers, and to determine the first protagonist fingerprint and the second protagonist fingerprint as text data, based on the sixth mapping relationship table.

[0183] The index relationship establishment module is used to establish index relationships between the first publishing object and the first protagonist fingerprint, the second publishing object and the second protagonist fingerprint, based on the text number and text data;

[0184] The protagonist fingerprint acquisition module is used to acquire the first protagonist fingerprint corresponding to the first publishing object and the second protagonist fingerprint corresponding to the second publishing object based on the established index relationship.

[0185] The similarity calculation module is used to calculate the similarity between the obtained fingerprints of the first protagonist and the second protagonist.

[0186] In one embodiment, a fingerprint-based association identification device is provided, further comprising a same publishing object determination module, used for:

[0187] Sort the objects according to their degree of association and generate a corresponding association degree sequence; determine the maximum degree of association based on the association degree sequence, and obtain the first and second publishing objects corresponding to the maximum degree of association, and determine them as the same publishing object.

[0188] In one embodiment, the same publishing object determination module is also used for:

[0189] Based on the preset correlation threshold and each correlation level, the correlation level that meets the judgment requirements is determined; the first publishing object and the second publishing object corresponding to the correlation level that meets the judgment requirements are determined as the same publishing object.

[0190] The modules in the aforementioned fingerprint-based identification device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0191] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 7As shown, the computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores data such as a first object number, a second object number, first video content data, second video content data, a first keyframe, a second keyframe, a first entity object, a second entity object, fingerprint data of the first object, fingerprint data of the second object, fingerprints of the first protagonist, fingerprints of the second protagonist, similarity, and the degree of association between the first and second publishing objects. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network connection. When executed by the processor, the computer program implements a protagonist fingerprint-based association identification method.

[0192] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0193] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0194] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0195] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0196] It should be noted that the information on the publishing targets involved in this application (including but not limited to the device information of the publishing target, the personal information of the publishing target, etc.) and data (including but not limited to the data used for analysis, the data stored, the data displayed, etc.) are all information and data authorized by the corresponding publishing targets or fully authorized by all parties, and the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0197] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0198] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0199] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for association and identification based on the protagonist's fingerprint, characterized in that, The method includes: Acquire video content data to be processed, the video content data including first video content data published by a first publishing object and second video content data published by a second publishing object; Entity recognition is performed based on the video content data to obtain the corresponding entity recognition result. The entity recognition result includes a first entity object corresponding to the first video content data and a second entity object corresponding to the second video content data. For each of the first entity objects and each of the second entity objects identified, content clustering processing is performed to generate corresponding first object fingerprint data and second object fingerprint data; Based on the protagonist recognition algorithm, the first protagonist fingerprint corresponding to the first publishing object is determined from the fingerprint data of each of the first objects, and the second protagonist fingerprint corresponding to the second publishing object is determined from the fingerprint data of each of the second objects. The degree of association between the first publishing object and the second publishing object is determined based on the similarity between the fingerprints of the first protagonist and the second protagonist.

2. The method according to claim 1, characterized in that, The entity recognition based on the video content data, to obtain the corresponding entity recognition result, includes: Extract the first keyframe corresponding to the first video content data and the second keyframe corresponding to the second video content data respectively; Entity object recognition is performed based on the first keyframe and the second keyframe to obtain a first entity object corresponding to the first keyframe and a second entity object corresponding to the second keyframe.

3. The method according to claim 2, characterized in that, The first video content data and the first keyframe, and the second video content data and the second keyframe, are associated and stored through a first mapping table; the step of performing content clustering processing on each identified first entity object and each second entity object to generate corresponding first object fingerprint data and second object fingerprint data includes: Obtain a first mapping table between the first video content data and the first keyframe, and between the second video content data and the second keyframe; Based on the first entity objects in each of the first keyframes, the second entity objects in each of the second keyframes, and the first mapping table, a second mapping table is established between the first video content data and each of the first entity objects, and between each of the second video content data and each of the second entity objects. Based on the second mapping table, clustering is performed on each of the first entity objects and each of the second entity objects at the content granularity level to generate corresponding first object fingerprint data and second object fingerprint data, and a third mapping table is established between the first video content data and the first object fingerprint data, and between the second video content data and the second object fingerprint data.

4. The method according to claim 3, characterized in that, The step of determining the first protagonist fingerprint corresponding to the first publishing object from the fingerprint data of each of the first objects, and determining the second protagonist fingerprint corresponding to the second publishing object from the fingerprint data of each of the second objects, based on the protagonist recognition algorithm, includes: Based on the aforementioned protagonist recognition algorithm, a protagonist recognition model that meets the recognition requirements is determined; According to the protagonist recognition model that meets the recognition requirements, protagonist recognition is performed on the fingerprint data of each first object and the fingerprint data of each second object. The first protagonist fingerprint corresponding to the first publishing object is determined from the fingerprint data of each first object, and the second protagonist fingerprint corresponding to the second publishing object is determined from the fingerprint data of each second object. The protagonist recognition model that meets the recognition requirements is obtained by training an initial human face recognition model. Based on the third mapping table, and the first protagonist fingerprint and the second protagonist fingerprint, a fourth mapping table is established between the first video content data and the first protagonist fingerprint, and between the second video content data and the second protagonist fingerprint.

5. The method according to claim 4, characterized in that, The first video content data and the first publishing object, as well as the second video content data and the second publishing object, are associated and stored through a fifth mapping relationship list; after establishing a fourth mapping relationship table between the first video content data and the first protagonist fingerprint, and the second video content data and the second protagonist fingerprint, based on the third mapping relationship table, the first protagonist fingerprint, and the second protagonist fingerprint, the method further includes: Obtain the first video content data and the first publishing object, as well as the fifth mapping table between the second video content data and the second publishing object; Based on the fourth and fifth mapping relationship tables, data fusion processing is performed on the first publishing object and the first protagonist fingerprint, and the second publishing object and the second protagonist fingerprint to generate a sixth mapping relationship table between the first publishing object and the first protagonist fingerprint, and the second publishing object and the second protagonist fingerprint.

6. The method according to claim 5, characterized in that, Before determining the degree of association between the first publishing object and the second publishing object based on the similarity between the first protagonist's fingerprint and the second protagonist's fingerprint, the method further includes: According to the sixth mapping relationship table, the first object number of the first publishing object and the second object number of the second publishing object are determined as text numbers, and the first protagonist fingerprint and the second protagonist fingerprint are determined as text data; Based on the text number and the text data, establish an index relationship between the first publishing object and the first protagonist fingerprint, and between the second publishing object and the second protagonist fingerprint; Based on the established index relationship, obtain the first protagonist fingerprint corresponding to the first publishing object and the second protagonist fingerprint corresponding to the second publishing object; Calculate the similarity between the obtained first protagonist fingerprint and second protagonist fingerprint.

7. The method according to claim 1 or 2, characterized in that, The method further includes: The correlation degree is sorted according to the correlation degree to generate a corresponding correlation degree sequence; Based on the correlation degree sequence, the maximum correlation degree is determined, and the first and second publishing objects corresponding to the maximum correlation degree are obtained and determined to be the same publishing object; or Based on the preset correlation threshold and each correlation degree, the correlation degree that meets the judgment requirements is determined; The first and second publishing objects that correspond to the degree of association that meets the determination criteria are identified as the same publishing object.

8. A fingerprint-based association identification device, characterized in that, The device includes: The video content data acquisition module is used to acquire video content data to be processed, wherein the video content data includes first video content data published by a first publishing object and second video content data published by a second publishing object; An entity recognition module is used to perform entity recognition based on the video content data to obtain corresponding entity recognition results. The entity recognition results include a first entity object corresponding to the first video content data and a second entity object corresponding to the second video content data. The protagonist identification module includes: an object fingerprint data generation unit, used to perform content clustering processing on each of the identified first entity objects and each of the second entity objects to generate corresponding first object fingerprint data and second object fingerprint data; and a protagonist fingerprint determination unit, used to determine the first protagonist fingerprint corresponding to the first publishing object from each of the first object fingerprint data and the second protagonist fingerprint corresponding to the second publishing object from each of the second object fingerprint data based on the protagonist identification algorithm. The association degree determination module is used to determine the association degree between the first publishing object and the second publishing object based on the similarity between the fingerprints of the first protagonist and the fingerprints of the second protagonist.

9. The apparatus according to claim 8, characterized in that, The entity recognition module includes: The keyframe extraction unit is used to extract the first keyframe corresponding to the first video content data and the second keyframe corresponding to the second video content data, respectively. The entity object generation module is used to identify entity objects based on the first keyframe and the second keyframe to obtain a first entity object corresponding to the first keyframe and a second entity object corresponding to the second keyframe.

10. The apparatus according to claim 9, characterized in that, The first video content data and the first keyframe, and the second video content data and the second keyframe, are associated and stored through a first mapping table; the object fingerprint data generation unit is further configured to: Obtain a first mapping relationship table between the first video content data and the first key frame, and between the second video content data and the second key frame; based on the first entity object in each of the first key frames, the second entity object in each of the second key frames, and the first mapping relationship table, establish a second mapping relationship table between the first video content data and each of the first entity objects, and between each of the second video content data and each of the second entity objects. Based on the second mapping table, clustering is performed on each of the first entity objects and each of the second entity objects at the content granularity level to generate corresponding first object fingerprint data and second object fingerprint data, and a third mapping table is established between the first video content data and the first object fingerprint data, and between the second video content data and the second object fingerprint data.

11. The apparatus according to claim 10, characterized in that, The protagonist fingerprint determination unit is also used for: Based on the aforementioned protagonist recognition algorithm, a protagonist recognition model that meets the recognition requirements is determined; according to the protagonist recognition model that meets the recognition requirements, protagonist recognition is performed on each first object fingerprint data and each second object fingerprint data, determining the first protagonist fingerprint corresponding to the first publishing object from each first object fingerprint data, and determining the second protagonist fingerprint corresponding to the second publishing object from each second object fingerprint data; the protagonist recognition model that meets the recognition requirements is obtained by training an initial human face recognition model; Based on the third mapping table, and the first protagonist fingerprint and the second protagonist fingerprint, a fourth mapping table is established between the first video content data and the first protagonist fingerprint, and between the second video content data and the second protagonist fingerprint.

12. The apparatus according to claim 11, characterized in that, The first video content data and the first publishing object, as well as the second video content data and the second publishing object, are associated and stored through a fifth mapping relationship list; the device further includes: The fifth mapping relationship table acquisition module is used to acquire the fifth mapping relationship table between the first video content data and the first publishing object, and between the second video content data and the second publishing object; The sixth mapping relationship table generation module is used to perform data fusion processing on the first publishing object and the first protagonist fingerprint, and the second publishing object and the second protagonist fingerprint, according to the fourth mapping relationship table and the fifth mapping relationship table, to generate a sixth mapping relationship table between the first publishing object and the first protagonist fingerprint, and the second publishing object and the second protagonist fingerprint.

13. The apparatus according to claim 12, characterized in that, The device further includes: a text number and text data determination module, used to determine the first object number of the first publishing object and the second object number of the second publishing object as text numbers according to the sixth mapping relationship table, and to determine the first protagonist fingerprint and the second protagonist fingerprint as text data; The index relationship establishment module is used to establish index relationships between the first publishing object and the first protagonist fingerprint, and between the second publishing object and the second protagonist fingerprint, based on the text number and the text data; The protagonist fingerprint acquisition module is used to acquire the first protagonist fingerprint corresponding to the first publishing object and the second protagonist fingerprint corresponding to the second publishing object based on the established index relationship. The similarity calculation module is used to calculate the similarity between the acquired first protagonist fingerprint and second protagonist fingerprint.

14. The apparatus according to claim 8 or 9, characterized in that, The device further includes a module for determining the same publishing target, used for: Sort the results according to the degree of association to generate a corresponding degree of association sequence; determine the maximum degree of association based on the degree of association sequence, and obtain the first publishing object and the second publishing object corresponding to the maximum degree of association, and determine them as the same publishing object; Alternatively, based on a preset association degree threshold and each of the aforementioned association degrees, an association degree that meets the judgment requirements can be determined; the first publishing object and the second publishing object corresponding to the association degree that meets the judgment requirements can be determined as the same publishing object.

15. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

Images