Video character recognition and copyright protection method based on context and facial information

Abstract

The application discloses a video role recognition and copyright protection method based on context and facial information, and the steps include: 1) clustering each frame of the video based on the scene semantic features of each frame, clustering frames with similar spatial layout and background into a class, and taking each frame of the same class as the context area of the corresponding scene; 2) dividing the video into continuous shots and identifying the role position of each role in each shot, and generating the trajectory of each role; 3) clustering the trajectories of the same role under different shots into a class, and the trajectories of different roles belong to different categories; 4) extracting the facial feature vector of each role and matching the facial feature vector with the facial feature vectors of roles in the original IP role database; 5) confirming the role label of the role according to the appearance features of the trajectories in the trajectory clustering of the same role; and 6) matching the role label of each role with the corresponding role in the original copyright database, and judging whether the video infringes the copyright.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, artificial intelligence and digital copyright protection, and specifically relates to a method combining context scene recognition and character recognition, which is used for character tracking and copyright protection of image, video and other content. BACKGROUND

[0002] With the explosive growth of digital media content, cartoon characters as the core visual assets in the film and television, game, animation and other industries frequently appear in social platforms, short video platforms and various digital creations. These cartoon characters usually have high visual recognition, distinctive style and strong symbolic nature, making them important carriers of IP economy. However, because of their wide dissemination and high recognition, the copyright problem of cartoon character images is increasingly serious, especially in unauthorized secondary creation, commercial embedding and AI-generated images, there are a large number of infringement behaviors.

[0003] The current mainstream video copyright protection methods mainly include digital fingerprint (such as perceptual hash) and digital watermarking technologies. Digital fingerprint encodes the content features of images or video frames to achieve duplicate detection, while digital watermarking embeds copyright information in visual content for tracking. However, these methods have significant limitations when faced with complex scenarios such as stylized processing, video editing, deformation imitation and AI redrawing. Image style transfer can destroy the pixel structure of the image, leading to hash mismatch; and watermark is often difficult to preserve after editing such as recompression and scaling, and cannot be used as a reliable copyright identifier.

[0004] In recent years, some research in academia has focused on cartoon character recognition, which can be used as a reference for copyright protection. For example, Liu et al. proposed a cartoon face recognition method based on domain adaptation (Cartoon Face Recognition via Domain Adaptation, ICME 2021), trying to narrow the domain gap between cartoon and real faces. However, this method significantly reduces the accuracy under stylized deformation; Zhou et al. proposed using face + pose trajectory for animated character re-identification (Character Re-identification in Animated Videos, ACM MM 2020), which improves the character matching ability in continuous shots to some extent, but does not model the context information. Although existing research has provided some preliminary solutions for cartoon character recognition, there is still a lack of a technical framework that can uniformly model the fusion of character context behavior, semantic scene consistency and facial features. Especially in complex copyright protection scenarios, the system needs not only to identify the specific identity of the character, but also to judge whether it matches the registered copyright image, and then to accurately locate and support the evidence of infringement behavior. SUMMARY

[0005] The present application aims at the problems of difficult identification of copyright infringement, inaccurate cross-scene identification and poor robustness of identification under style change of cartoon characters in the prior art, and proposes a video character identification and copyright protection method combining context semantic modeling and cartoon face identification. The method uses three information sources of scene semantic clustering, trajectory modeling and cartoon face identification for joint modeling, can realize stable identification of character identity in high stylization videos such as animation and science fiction, and further provides a technical basis for copyright ownership judgment.

[0006] The technical scheme of the present application is as follows:

[0007] A video character identification and copyright protection method based on context and face information, the steps of which include:

[0008] 1) Using an image scene identification model to extract the scene semantic features of each frame in the video; then clustering the frames of the video based on the scene semantic features of the frames, clustering the frames with similar spatial layout and background into a class, obtaining several scenes with consistent semantics; regarding the frames of the same class as the context area of the corresponding scene;

[0009] 2) dividing the video into continuous shots and identifying the character positions of each character in each shot, performing time sequence tracking on each character, and generating the trajectory of each character;

[0010] 3) using multiple continuous shots in the same scene to construct a cross-shot character graph; wherein each node in the character graph represents a trajectory in the same shot, and each edge represents the co-occurrence relationship, visual similarity or time sequence connection between two trajectories in the same context area in two shots in the same scene; then using a graph neural network to perform embedding learning on the character graph to obtain the embedding representation of each node; clustering the trajectories according to the embedding representation of the nodes, and clustering the trajectories of the same character in different shots into a class, and the trajectories of different characters belong to different categories;

[0011] 4) extracting the face feature vector of each character in each frame where the face region can be detected; matching the extracted face feature vector with the face feature vectors of each character in the original IP character database to obtain the preliminary matching result of the character identity corresponding to the extracted face feature vector;

[0012] 5) Extract the appearance features of each trajectory in the same role trajectory cluster as the appearance features of the role, and calculate the confidence of the role according to the appearance features of the role, the appearance feature vector of the role in the original IP role database, and the appearance feature vectors of the trajectory clusters of other roles in the same context. If the confidence is greater than a set threshold, the role label of the role is confirmed. The basic logic is to calculate the similarity f of the appearance features of the role and the appearance feature vector of the role in the original IP role database, and calculate the maximum value max f' of the similarity of the appearance features of the role and the appearance feature vectors of the trajectory clusters of other roles in the same scene and the same time period. Then the confidence of the role is calculated

[0013] θ = f - max f';

[0014] 6) Match the role label of each role with the corresponding role in the original copyright database. If the matched role is not authorized for the video, mark the video as suspicious infringement.

[0015] Preferably, the weight of the edge between two nodes in the role graph is determined based on the spatial distance, appearance similarity, co-occurrence frequency, and behavior consistency features between the two nodes.

[0016] Preferably, the spatial distance is the position coincidence degree of two trajectories in the scene; the appearance similarity is the cosine similarity of the color distribution, body features, and facial features of the roles in the two trajectories; the co-occurrence frequency is the co-occurrence statistics of the two trajectories between different shots; the behavior consistency is whether the actions of the roles in the two trajectories are continuous in the shots; and the weighted average of the spatial distance, appearance similarity, co-occurrence frequency, and behavior consistency features between the two nodes is used as the weight of the edge between the two nodes.

[0017] Preferably, a cartoon role detection network is used to identify the position of the role in each shot; a multi-target tracking method is used to track the role in time sequence, generating the trajectory of each role.

[0018] Preferably, a SOTA video shot segmentation algorithm is used to divide the video into continuous shots.

[0019] Preferably, an image scene recognition model is used to extract the scene semantic features of each frame.

[0020] Preferably, the trajectory of the role includes the position, size, color histogram, and inter-frame movement vector of the role in the shot.

[0021] Preferably, a cartoon-specific face recognition model is used to extract the facial feature vector of each role.

[0022] A server, comprising a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program comprising instructions for performing the above method.

[0023] A computer readable storage medium having stored thereon a computer program, wherein the computer program, when executed by a processor, implements the above method.

[0024] The present application mainly includes the following contents:

[0025] 1. Scene recognition and semantic clustering

[0026] Using an image scene recognition model, such as a visual feature encoder based on ResNet50+NetVLAD, to perform frame-level analysis on the video and extract the scene semantic features of each frame. Based on the frame-based scene semantic features, clustering methods such as DBSCAN or KMeans are used to cluster the frames of the video, frames with similar spatial layout and background are clustered into a class, and scenes with consistent semantics are identified, such as "Spring Day Department's living room" and "school classroom" in "Crayon Shin-chan". In actual operation, first perform shot boundary detection, then extract scene features from the middle frames in the shot. Each frame in the same cluster (i.e. each frame belonging to the same scene) is taken as a scene unit, and each scene unit is recorded as a "context area" as the context basis for subsequent role identity reasoning.

[0027] 2. Shot segmentation and role trajectory construction

[0028] Using SOTA video shot segmentation algorithms such as PySceneDetect to divide the video into continuous shots. In each shot, use a cartoon character detection network such as Cartoon-YOLO based on YOLOv5 to identify the position of each character in the shot, and use multi-target tracking methods such as DeepSort to track the characters in time sequence, generating the trajectory of each character. Each character's trajectory contains information such as the position, size, color histogram, and inter-frame movement vector of the character in the shot.

[0029] 3. Contextual role graph modeling

[0030] For multiple consecutive shots in the same scene, a cross-shot role graph is constructed. Each node in the role graph represents a trajectory of a role in the same shot, and each edge represents the co-occurrence relationship, visual similarity or temporal connection between two trajectories in the same context area in two shots in the same scene. The edge weight is constructed based on the spatial distance, appearance similarity, co-occurrence frequency and behavior consistency features. The spatial distance uses the position coincidence of the trajectories of the two roles in the scene, the appearance similarity uses the cosine similarity of the color distribution, body features and face features in the trajectories of the two roles, the co-occurrence frequency uses the co-occurrence statistics of the two trajectories in different shots, and the behavior consistency mainly judges whether the actions (pose estimation) of the two roles in the shots are continuous. The edge weight is the weighted average of the above features. The graph neural network (such as GCN) is used to perform embedding learning on the role graph to obtain the embedding representation of each node; and the trajectories of the roles are clustered according to the embedding representation of the nodes, so that the trajectories of the same roles in different shots are clustered into one class, and the trajectories of different roles belong to different classes.

[0031] 4. Cartoon face feature extraction and matching

[0032] In each frame where a face region can be detected, a cartoon-specific face recognition model, such as CartoonFaceNet based on ArcFace or FaceNet, is used to extract a face feature vector for each role. The model is pre-trained on a cartoon face dataset, such as WebCaricature or AnimeFace, to enhance its ability to recognize non-real faces. The extracted face feature vector is matched with the face feature vectors of each role in the original IP role database (using Euclidean distance or FAISS index structure) to obtain a preliminary matching result of the corresponding role identity of the extracted face feature vector. The preliminary matching result is used as a key feature of appearance similarity for clustering in the context graph to improve clustering accuracy.

[0033] 5. Joint inference and role identification

[0034] The appearance features of different trajectories in the context graph clustering, especially the face features, are extracted, and a weighted voting or confidence fusion method is used to unify the role identification. For example, for a certain cluster category, the corresponding role is identified by integrating the face and behavior features in the context graph, and the similarity with other categories in the same context is considered to calculate the confidence score, and the most likely role identity is selected. The confidence score is positively correlated with the feature similarity of the corresponding role and negatively correlated with the similarity of other categories to avoid misidentifying different roles in the same scene as the same role. A role label credibility threshold is set, and only the results exceeding the threshold are output as "determined role", and the rest are in the "to be confirmed" state for manual review.

[0035] 6. Copyright comparison and infringement detection

[0036] The identified character information is compared with the corresponding characters in the original copyright database (which may include character images, action templates, scene appearance records, etc.). For detected character appearance segments, the timestamp, scene number, and identity information are recorded, and a structured report is output. If a character appears in an unauthorized video segment (such as detecting illegal use of the "Crayon Shin-chan" character image), it is marked as suspected infringement, and the original evidence path and comparison results are provided to support subsequent rights protection or removal.

[0037] Compared with the prior art, the positive effects of the present invention are as follows:

[0038] This invention provides a video character recognition method that combines contextual semantic modeling, cartoon facial recognition, and trajectory reasoning. It supports accurate identification and copyright attribution determination of cartoon characters across shots, scenes, and styles. This invention possesses strong robustness, scalability, and copyright explainability, and can meet the copyright protection needs of animation and science fiction IP content in the contemporary digital ecosystem. Attached Figure Description

[0039] Figure 1 This is a flowchart of the role recognition and copyright protection method of the present invention. Detailed Implementation

[0040] To more clearly illustrate the implementation process of this invention, the following example uses a suspected infringing animated video uploaded to a video platform to demonstrate how to identify the character "Nohara Shinnosuke" from the well-known animated IP "Crayon Shin-chan" and determine copyright ownership. The video clip is approximately 90 seconds long, with AI-style redrawn visuals, including multiple camera cuts and scene changes, and does not include character name tags.

[0041] The various features and algorithms used in this invention are not limited to the specific algorithms selected below; the use of specific algorithms is only for illustrative purposes.

[0042] 1. Video preprocessing and shot segmentation

[0043] The original video was read at 30 frames per second. PySceneDetect was used to segment the footage, resulting in N = 17 footage segments, denoted as ${S1,S2,…,S$. 17 After the shots are segmented, the middle frame of each shot is selected as the representative frame, and its scene semantic features are extracted.

[0044] 2. Scene Recognition and Clustering

[0045] For each representative frame, the global scene feature vector is extracted using the ResNet-50+NetVLAD architecture. All scene features constitute a set Then, the DBSCAN clustering algorithm is used to cluster these scene vectors, ultimately forming M=3 semantically consistent scene clusters, corresponding to environments such as "living room", "kitchen", and "bedroom".

[0046] 3. Cartoon Character Detection and Trajectory Extraction

[0047] Cartoon characters were detected using the Cartoon-YOLOv5 model across all video frames. This model is a finely tuned version of YOLOv5 on cartoon character datasets (such as AnimeFace200k), supporting the detection of head and full-body bounding boxes for cartoon characters. The detection results for each frame were then used to perform trajectory tracking using the DeepSort algorithm, resulting in a trajectory sequence T for each character. i ={(x t ,y t ,w t ,h t ) | t∈[t1,t2]}. Where, (x t ,y t ,w t ,h t () represents the position and size of the detection box for the character in frame t. Finally, K trajectories {T1, T2, ..., T} of the character are obtained. K}

[0048] 4. Context Graph Construction

[0049] Construct a graph structure G = (V, E) between the trajectories of the characters, where each node v i ∈V represents a trajectory T i , side e ij ∈E represents the trajectory T i and T j The degree of correlation between them. The edge weight is defined as follows: w ij =α·sim app (T i ,T j )+β·sim spatial (T i ,T j )+γ·sim co-occur (T i ,T j ). Among them, sim app Sim represents the cosine distance between appearance similarities (including facial features, behavioral features, color distribution features, etc.). spatial sim represents the average positional overlap. co-occurThis represents the probability of co-occurrence within the same scene cluster. The coefficients are set to α = 0.4, β = 0.3, and γ = 0.3, obtained through cross-validation. A graph neural network (GCN) is used to learn the representation of each node on this graph, performing identity clustering between trajectories.

[0050] 5. Cartoon facial recognition and fusion

[0051] In frames where faces can be detected, facial feature vectors are extracted using the CartoonFaceNet model. This model was modified from ArcFace and obtained through transfer learning on WebCaricature and CartoonSet. The extracted features were then compared with standard role vectors from the IP role database. To perform the matching, cosine similarity is calculated:

[0052]

[0053] For each character's trajectory clustering category in the context, facial features of keyframes (frames containing the character's face) in each trajectory are calculated, and the above matching is performed, assuming the result is "Nohara Shinnosuke" and a cosine similarity f. Simultaneously, the similarity f' between other categories matching "Nohara Shinnosuke" is calculated. The two calculations are combined to obtain the final confidence score, such as confidence θ = f - max f'. If the score is higher than the set threshold θ = 0.82, the face is considered to belong to "Nohara Shinnosuke". This result is used as a strong supervisory signal and fused with the trajectory clustering result. The final character identification strategy is: 1) If the various features of the trajectory clustering are consistent with the facial recognition result (during trajectory clustering, facial features are used for matching; at the same time, it is checked whether various features are consistent with the recognition result), the value is directly assigned; 2) If they are inconsistent, a voting fusion strategy is adopted, and the weights are recalculated according to the confidence level to obtain the final character label.

[0054] 6. Copyright comparison and infringement detection

[0055] The identified "Shin-chan Nohara" character's trajectory spanned multiple shots, appearing at times of [12.3s, 20.1s], [41.6s, 49.8s], and [70.0s, 82.5s], corresponding to the scenes of "living room," "kitchen," and "bedroom," respectively. Comparison of these clips with standard videos in the copyright holder's license database revealed no corresponding license records for these segments. Furthermore, the cartoon's facial features highly matched the IP standard library, leading to the conclusion of suspected copyright infringement.

[0056] The system automatically generates an infringement report, including:

[0057] Character Name: Nohara Shinnosuke;

[0058] Video timestamp ranges: [12.3–20.1s], [41.6–49.8s], [70.0–82.5s];

[0059] Similarity score: 0.894;

[0060] Scene location labels: Scene_1, Scene_2, Scene_3;

[0061] Judgment: High-confidence infringement;

[0062] Screenshot of the chain of evidence (automatically export several keyframe images).

[0063] Although specific embodiments of the invention have been disclosed for illustrative purposes and to aid in understanding and implementing the invention, those skilled in the art will understand that various substitutions, variations, and modifications are possible without departing from the spirit and scope of the invention and the appended claims. Therefore, the invention should not be limited to the content disclosed in the preferred embodiments, and the scope of protection claimed by the invention is defined by the claims.

Claims

1. A method for video character recognition and copyright protection based on context and facial information, comprising the following steps: 1) Extract scene semantic features from each frame of the video using an image scene recognition model; Then, based on the scene semantic features of the frames, the frames of the video are clustered together, and frames with similar spatial layout and background are grouped into one class to obtain several semantically consistent scenes. Frames of the same type are used as the context area of ​​the corresponding scene; 2) Divide the video into continuous shots and identify the position of each character in each shot. Perform time-series tracking on each character to generate the trajectory of each character. 3) Construct a cross-shot character atlas using multiple consecutive shots within the same scene; where each node in the character atlas represents a trajectory within the same shot, and each edge represents the co-occurrence relationship, visual similarity, or temporal connection between two trajectories in two shots within the same scene within the same context area; then, use a graph neural network to perform embedding learning on the character atlas to obtain the embedding representation of each node; cluster the trajectories based on the node embedding representation, grouping the trajectories of the same character in different shots into one category, while the trajectories of different characters belong to different categories; wherein, based on the character In the graph, the spatial distance, appearance similarity, co-occurrence frequency, and behavioral consistency features between two nodes determine the weight of the edge between them. The spatial distance is the degree of overlap between the positions of the two trajectories in the scene. The appearance similarity is the cosine similarity of the color distribution, body features, and facial features of the characters in the two trajectories. The co-occurrence frequency is the co-occurrence statistics of the two trajectories in different shots. The behavioral consistency is whether the actions of the characters in the two trajectories are continuous in the shot. The weight of the edge between the two nodes is obtained by weighting the spatial distance, appearance similarity, co-occurrence frequency, and behavioral consistency features. 4) In each frame where the facial region can be detected, extract the facial feature vector of each character; match the extracted facial feature vector with the facial feature vectors of each character in the original IP character database to obtain the preliminary matching result of the character identity corresponding to the extracted facial feature vector. 5) Extract the appearance features of each trajectory in the trajectory cluster of the same role as the appearance features of the role. Based on the appearance features of the role, the appearance feature vector of the role in the original IP role database, and the appearance feature vectors of other role trajectory clusters in the same context, calculate the confidence of the role. If the confidence is greater than the set threshold, the role label of the role is confirmed. 6) Match the character tag of each character with the corresponding character in the original copyright database. If the matched character has not authorized the video, mark the video as suspected infringement.

2. The method according to claim 1, characterized in that, A cartoon character detection network is used to identify the position of characters in each shot; a multi-target tracking method is used to perform temporal tracking of the characters and generate the trajectory of each character.

3. The method according to claim 1, characterized in that, The video is divided into consecutive shots using the SOTA video shot segmentation algorithm.

4. The method according to claim 1, characterized in that, The semantic features of the scene in each frame are extracted using an image scene recognition model.

5. The method according to claim 1, characterized in that, The character's trajectory includes the character's position, size, color histogram, and inter-frame movement vector within the shot.

6. The method according to claim 1, characterized in that, The facial feature vector of each character is extracted using a cartoon-specific facial recognition model.

7. A server, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 6.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 6.

Images