Method, device and equipment for information association of off-line video and storage medium

Abstract

The application relates to the technical field of data analysis, and provides a method and device for information correlation of offline videos, equipment and a storage medium. The method comprises the following steps: preprocessing single-set videos, obtaining a maximum face area graph and a maximum single-face original graph of each type of face in a key frame; performing global clustering based on the maximum face area graph and the maximum single-face original graph of each type of face extracted from all single-set videos; selecting a global target face-removing large graph and a global target maximum single-face original graph of each type of face from the global clustering result; and calling an external star face recognition service for identity recognition based on the global target face-removing large graph and / or the global target maximum single-face original graph, and associating each recognition object with real information and role information. The application performs identity recognition after hierarchical processing, key frame screening and representative image selection, can reduce the data volume of global processing, and improves the processing speed and accuracy.

Description

Technical Field

[0001] This application relates to the field of data analysis technology, and in particular to methods, apparatus, devices and storage media for information association of videos not yet online. Background Technology

[0002] In the construction of multimodal video datasets, intelligent media asset management, and automated content analysis of long videos (such as TV series and movies), current methods for facial recognition of already released videos typically rely on publicly available metadata for retrieval and matching, such as cast lists and stills. For long videos that are not yet released, the lack of publicly available online information makes it impossible to directly associate characters.

[0003] Current technologies typically employ end-to-end face clustering algorithms, directly processing massive amounts of video frames. However, long videos (such as TV series with dozens of episodes) have an enormous number of frames, causing the computational load for direct global clustering to increase exponentially, resulting in wasted computational resources and low efficiency. Furthermore, videos often contain numerous passersby, extras, or blurred profiles; direct identification without filtering leads to a large amount of noisy data, affecting the accuracy of identifying the main character. Moreover, for unreleased new dramas, the lack of public information makes it difficult to distinguish between actor names and real names, hindering identity mapping.

[0004] It is evident that when performing automated content analysis on long videos that are not yet online to identify role-related information, there are issues with low efficiency and accuracy in information association. Summary of the Invention

[0005] This application provides a method, apparatus, device, and storage medium for information association of videos not yet online, in order to solve the technical problem of low efficiency and accuracy in information association when performing automated content analysis on long videos not yet online to identify role-related information.

[0006] According to one aspect of the embodiments of this application, this application provides an information association method for videos not yet released online. The method includes: preprocessing a single episode of video to obtain the maximum facial region image and the maximum single face original image for each type of face in keyframes; performing global clustering based on the maximum facial region image and the maximum single face original image for each type of face extracted from all single episodes of video, and selecting a global target face cutout image and a global target maximum single face original image for each type of face from the global clustering results; and calling an external celebrity face recognition service to perform identity recognition based on the global target face cutout image and / or the global target maximum single face original image, and associating each recognized object with real information and role information.

[0007] Optionally, the step of preprocessing the single-episode video to obtain the maximum facial region image and the maximum single face original image for each type of face in the keyframes includes: decomposing the long video to be identified into multiple single-episode videos; performing face detection and feature extraction on each single-episode video, and filtering the keyframes based on the extracted facial features; constructing a facial feature vector based on the facial features in the keyframes of the single-episode video, and selecting the maximum facial region image and the maximum single face original image for each type of face based on the facial feature vector.

[0008] Optionally, the step of performing face detection and feature extraction on each individual video episode, and filtering the keyframes based on the extracted face features, includes: reading the individual video episode file, obtaining bounding boxes and key points in each image frame using a face detection algorithm; determining the face positions in each image frame based on the bounding boxes and key points, and obtaining the number of faces; filtering the keyframes based on the number of faces in each image frame, and marking image frames that include faces and whose number of faces is lower than a preset face number threshold as the keyframes.

[0009] Optionally, the step of performing face detection and feature extraction on each individual video episode, and filtering the keyframes based on the extracted face features, includes: reading the individual video episode file, obtaining the bounding box and key points in each image frame using a face detection algorithm; identifying the face position in each image frame based on the bounding box and key points in each image frame; analyzing the face position, and filtering image frames with facial occlusion and / or image quality that does not meet a preset quality threshold, so as to retain the keyframes.

[0010] Optionally, the step of constructing a facial feature vector based on the facial features in the keyframes of a single video episode, and selecting the maximum facial region image and the maximum single face original image for each type of face based on the facial feature vector, includes: for each single video episode, based on the facial positions extracted from the keyframes, extracting the facial features of each face using a preset facial feature recognition model to construct the facial feature vectors respectively, and calculating the similarity between each facial feature vector; based on the similarity between each facial feature vector, classifying the faces in the single video episode whose similarity exceeds a preset similarity threshold to obtain multiple single video face sets; and selecting the maximum facial region image and the maximum single face original image for each type of face from each single video face set.

[0011] Optionally, the step of performing global clustering based on the maximum facial region image and the maximum single face original image extracted from all single-episode videos for each type of face, and selecting the global target face-cutting image and the global target maximum single face original image for each type of face from the global clustering results, includes: comparing the maximum facial region image and the maximum single face original image selected from all single-episode videos using a similarity algorithm, and globally aggregating them to the same global ID to obtain the global clustering result, wherein the global clustering result includes the global ID corresponding to each type of face; and selecting the global target face-cutting image and the global target maximum single face original image from the global ID corresponding to each type of face based on preset filtering conditions.

[0012] Optionally, the step of calling an external celebrity face recognition service for identity verification based on the global target face cutout image and / or the global target maximum single face original image, and associating the real information and role information of each identified object, includes: converting the format of the global target face cutout image and / or the global target maximum single face original image according to the data input requirements of the external celebrity face recognition service; initiating an interface call request through a preset communication protocol based on the format-converted global target face cutout image and / or the global target maximum single face original image to request the external celebrity face recognition service for identity matching; and obtaining the return result based on the external celebrity face recognition service, wherein the return result includes the role information, the real information, and the mapping relationship between the role information and the real information.

[0013] According to another aspect of the embodiments of this application, this application provides an information association device for unreleased videos. The device includes: an image acquisition module, used to preprocess a single episode of video to acquire the maximum facial region image and the maximum single face original image of each type of face in keyframes; a face clustering module, used to perform global clustering based on the maximum facial region image and the maximum single face original image of each type of face extracted from all single episodes of video, and select the global target cutout image and the global target maximum single face original image of each type of face from the global clustering results; and an identity recognition module, used to call an external celebrity face recognition service to perform identity recognition based on the global target cutout image and / or the global target maximum single face original image, and associate each recognized object with real information and role information.

[0014] According to another aspect of the embodiments of this application, this application provides an electronic device, including: a processor, a memory, and a network interface. The memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor communicates with the memory through the network interface, and the processor executes the machine-readable instructions to perform the steps of the information association method for unreleased videos.

[0015] According to another aspect of the embodiments of this application, this application provides a storage medium having processor-executable non-volatile program code that causes the processor to perform the steps of the information association method for offline videos.

[0016] Compared with related technologies, the technical solutions provided in this application have the following advantages: The information association method for unreleased videos provided in this application fills the gap in character recognition for new dramas without internet metadata support by associating information between unreleased videos and data without public data. It can automatically discover and establish associations between real information and character information without manual pre-annotation. By hierarchically processing the process, breaking down long videos into multiple individual videos, performing facial processing on each individual video first, and then merging the facial processing results of all individual videos, high cohesion can be achieved within each individual video, reducing the amount of data processed globally and avoiding direct processing of massive amounts of video frames. The computational explosion caused by global clustering is significantly reduced by minimizing the data dimensionality of subsequent merging processing, thus improving processing speed. Furthermore, by selecting keyframes and extracting the largest regional image and the largest single face image for each face class from the keyframes as the basis for subsequent clustering, images with less correlation can be effectively filtered out, focusing on the main figures and significantly improving the purity of subsequent feature extraction and clustering accuracy. In addition, retaining the largest single face image allows for the use of richer contextual information, significantly improving the efficiency and accuracy of identity recognition when calling external celebrity face recognition services. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without any creative effort.

[0018] Figure 1 This is a schematic diagram of the hardware environment for an optional method for associating information about videos that are not online, according to an embodiment of this application. Figure 2 This is a schematic flowchart of an optional method for associating information about videos that are not currently online, according to an embodiment of this application. Figure 3 This is a schematic diagram of an optional process for step S202 provided in an embodiment of this application; Figure 4This is a flowchart illustrating an optional method for associating information about videos that are not currently online, according to an embodiment of this application. Figure 5 This is a structural diagram of an optional information association device for offline videos provided according to an embodiment of this application; Figure 6 This is a schematic diagram of an optional electronic device structure provided in an embodiment of this application. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] To address the problems mentioned in the background art, according to one aspect of the embodiments of this application, an embodiment of a method for associating information about videos that are not online is provided.

[0021] like Figure 1 As shown, the above-described method for associating information about videos not yet uploaded can be applied to, for example... Figure 1 The hardware environment shown is described. The system architecture 100 of the hardware environment includes a terminal device 101 and a server 103. The server 103 is connected to the terminal 101 via a network and can provide services to the terminal device 101 or clients installed on the terminal device 101. A database 105 can be set up on or independently of the server 103 to provide data storage services for the server 103. The network can include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0022] Users can use terminal device 101 to interact with server 103 via a network to receive or send messages. Various communication client applications can be installed on terminal device 101, such as web browsers, search applications, and instant messaging tools. Terminal device 101 can be various electronic devices with a display screen that support web browsing, including but not limited to smartphones, tablets, e-book readers, MP3 players (Moving Picture Experts Group Audio Layer III), MP4 players (Moving Picture Experts Group Audio Layer IV), laptops, and desktop computers. Server 103 can be a server providing various services, such as a backend server supporting the pages displayed on terminal device 101.

[0023] It should be noted that the information association method for unreleased videos provided in this application is generally executed by a server and / or terminal device, and correspondingly, the information association system for unreleased videos is generally set up in the server / terminal device. Furthermore, it should be understood that... Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0024] like Figure 2 As shown, Figure 2 This is a flowchart illustrating a method for associating information about videos not yet uploaded, provided in an embodiment of the present invention. Taking the method for associating information about videos not yet uploaded as being executed by a server as an example, the method includes the following steps: Step S202: Preprocess the single-episode video to obtain the maximum facial region image and the maximum single face original image for each type of face in the keyframes.

[0025] In this embodiment, the provided method for associating information with unreleased videos is applicable to the construction of multimodal video datasets, intelligent media asset management, and automated content analysis of long videos, such as TV series and movies. Particularly important is the accurate matching of faces in videos with character names in scripts and real-life actor names when constructing a multimodal dataset for training a "dialogue-speaker" recognition model.

[0026] The maximum facial region image refers to the image with the largest facial area among similar faces identified from at least one keyframe selected from a single video episode. Then, face matting is performed on the corresponding image frame to obtain the maximum matted face image. Obtaining the maximum facial region image ensures the clarity of face recognition, facilitates accurate calculation of face similarity, and enables global clustering.

[0027] The largest single-face original image refers to the original image with the largest face area among at least one keyframe selected from a single episode of video, without any image cutout. By extracting not only the largest facial region image but also retaining the largest single-face original image, richer contextual information can be utilized, significantly improving the recall and accuracy of the celebrity recognition interface.

[0028] In some examples, a complete long video can be broken down into multiple individual videos, and each individual video can be preprocessed. The preprocessing may include enhancing the image, performing frame-by-frame face detection and recognition, filtering video frames to retain keyframes, classifying faces in the individual videos based on the keyframes, and extracting the maximum facial region image and the maximum single face original image for each type of face.

[0029] In this embodiment, after the above preprocessing, the maximum facial region image and the maximum single-face original image for each type of face are selected from the keyframes of each video episode. For example, the maximum facial region image and the maximum single-face original image for face a, face b, and face c. After processing all the video episodes, multiple maximum facial region images and multiple maximum single-face original images for each type of face are obtained. For example, 20 maximum facial region images and 20 maximum single-face original images for face a, and 30 maximum facial region images and 30 maximum single-face original images for face b. To improve the accuracy of face similarity calculation, further image filtering can be performed on the multiple maximum facial region images and multiple maximum single-face original images for each type of face for global clustering.

[0030] Step S204: Perform global clustering based on the maximum facial region image and the maximum single face original image of each type of face extracted from all single-episode videos, and select the global target face cutout image and the global target maximum single face original image of each type of face from the global clustering results.

[0031] In this embodiment, face recognition and image extraction from individual video episodes are considered as video-level operations, while global clustering of all individual video episodes is considered as album-level operations. This application achieves a two-level cascaded clustering optimization architecture. Specifically, at the album level, global clustering is performed on the maximum facial region images and the largest single-face original image of each face type across all individual video episodes. Clustering can be achieved through similarity calculation. Thus, the maximum facial region image and the largest single-face original image retained for each face type across all individual video episodes are the global clustering results. In the global clustering results, each face type is treated as a dataset, including a subset obtained from clustering based on the maximum facial region images of that face and a subset obtained from clustering based on the largest single-face original images of that face. These are treated as separate subsets for subsequent data filtering.

[0032] Furthermore, the global clustering results for each face type include multiple maximum facial region images and the largest single-face original image. For example, if 5 maximum facial region images and 5 largest single-face original images are selected for each episode of a video series, and the entire series has 30 episodes, then the final global clustering results will include 150 maximum facial region images and 150 largest single-face original images. To reduce the data computation load for subsequent identity recognition and improve information association efficiency, this embodiment selects the final global target face-out image and the global target largest single-face original image for identity recognition from the global clustering results for each face type. This can be one image or multiple images.

[0033] Step S206: Based on the global target face cutout image and / or the global target maximum single face original image, call the external celebrity face recognition service to perform identity recognition, and associate the real information and role information of each recognized object.

[0034] The external celebrity face recognition service can include existing celebrity recognition interfaces provided by different vendors, such as Daxingluo. Upon successful call to the model recognition interface, the system will query the external celebrity knowledge base based on the selected global target face-out image and / or the largest single-face original image of the global target, and return the query results. These results include, but are not limited to, the mapping relationship between the identified object's role information and real information, similarity, confidence level, and timestamps. The real information of the identified object includes its real name, and the role information includes its role name. The mapping relationship between role information and real information includes the mapping between the identified object's real name and role name. For example, for identified object A, the mapping relationship between role information and real information is "Rolar-Liu Hua". This allows for the association of real information and role information based on faces for videos that are not yet available online, eliminating the problem of not being able to obtain the relationship between role information and real information due to a lack of publicly available data.

[0035] In this embodiment of the invention, by associating information with unreleased videos without publicly available data, the gap in character recognition for new dramas without internet metadata support is filled. This automatically discovers and establishes associations between real information and character information, eliminating the need for manual pre-annotation. By hierarchically processing the process, long videos are broken down into multiple individual videos. Face processing is performed on each individual video first, and then the face processing results from all individual videos are merged. This allows for high cohesion within each individual video, reducing the amount of data processed globally and avoiding the computational explosion problem caused by directly performing global clustering on massive video frames. This significantly reduces the data dimensionality of subsequent merging processing and improves processing speed. Furthermore, by selecting keyframes and extracting the largest regional image and the largest single-face original image for each type of face from the keyframes as the basis for subsequent clustering, images with lower relevance can be effectively filtered out, focusing on key characters and significantly improving the purity of subsequent feature extraction and clustering accuracy. In addition, retaining the largest single-face original image allows for the use of richer contextual information, significantly improving the efficiency and accuracy of identity recognition when calling external celebrity face recognition services.

[0036] In some alternative embodiments, combined with Figure 3 As shown, step S202 above includes: S2021, the long video to be identified is decomposed into multiple single-episode videos; S2022, Perform face detection and feature extraction on each individual video episode, and filter the key frames based on the extracted face features; S2023, construct a face feature vector based on the face features in the key frames of a single episode video, and select the maximum facial region image and the maximum single face original image for each type of face based on the face feature vector.

[0037] In some examples, long videos can be manually split using video editing tools such as CapCut, Premiere Pro, and VLC Media Player. Alternatively, automated splitting tools can be used for batch splitting, such as FFmpeg or Python+OpenCV. The splitting unit is based on a single episode. The intro and outro of each episode can be cut off to avoid selecting the same image frames from each episode's intro and outro, reducing subsequent computation and minimizing interference from the intro and outro.

[0038] In some examples, decomposition is performed using automated splitting tools, including: obtaining the standard duration of each episode, where each episode has a consistent duration; and performing batch splitting using FFmpeg based on the standard duration of each episode. For example, if the duration of a single episode video is 45 minutes, the FFmpeg tool is used to split the video files to be identified, with 45-minute intervals as the decomposition nodes.

[0039] Furthermore, face detection and recognition algorithms can be used to detect and recognize all faces in a single episode of video, extract the features of the detected faces, and then select keyframes based on the extracted facial features. These facial features include the face region, eyes, nose, ears, mouth, and accessories worn on the face, such as glasses and masks. Among all image frames in a single episode of video, keyframes refer to image frames where facial features can be captured more clearly. Keyframes can be selected by setting filtering conditions, including but not limited to image frame clarity, face clarity, number of faces, and image frame brightness.

[0040] Furthermore, after selecting keyframes, facial feature vectors are constructed based on the facial features of each face in the extracted keyframes of a single episode of the video. For example, in the first episode of the video, 20 keyframes are selected, containing a total of 40 faces. Facial feature vectors corresponding to these 40 faces are then constructed based on their facial feature distribution. Subsequently, similarity comparison calculations are used to cluster faces of the same type in the single episode of the video. Then, the largest facial region image and the largest single-face original image for each type of face are extracted by comparison. The largest facial region image can be used to analyze facial feature details, while the largest single-face original image includes hairstyle, clothing, background, etc., which is beneficial for anti-occlusion recognition.

[0041] In this embodiment, the processing of the long video set to be identified is broken down into two stages: single-video processing and full-video merging. This allows for keyframe filtering and face clustering within a single video, reducing the amount of data processed globally and avoiding the computational explosion caused by directly performing global clustering on massive video frames, thus improving processing speed. Furthermore, keyframe filtering effectively removes background noise and small-faced extras, focusing on key characters and significantly improving the purity and clustering accuracy of subsequent feature extraction. In addition, retaining the largest facial region image and the largest single-face original image for each face type, combined with image quality and semantic information, significantly improves the recall and accuracy of subsequent calls to external celebrity face recognition services.

[0042] In some optional embodiments, step S2022 above includes: Read a single video file and use a face detection algorithm to obtain the bounding box and key points in each image frame; The face positions in each image frame are determined based on the bounding box and the key points, and the number of faces is obtained. The keyframes are filtered based on the number of faces in each image frame, and image frames that include faces and whose number of faces is less than a preset face count threshold are marked as keyframes.

[0043] In some examples, a single video file is read, and the SCRFD (Sample and Computation Redistribution for Efficient Face Detection) algorithm is used to detect faces, obtaining bounding boxes (bboxes) and keypoints (kpss) in each frame. The bounding boxes can be used to locate face regions, and the keypoints can be used to construct vectors and calculate face similarity.

[0044] In some examples, bounding boxes are detected first, and then affine transformations are used to align the face to a standard template, such as a 112×112 pixel frontal face size, ensuring that key points such as the eyes, nose, and mouth fall on fixed coordinates. The image is then normalized, including pixel value normalization, illumination normalization, and color normalization, to facilitate the extraction of more accurate key points. From the aligned and normalized face image, unique biometric features are extracted, possessing semantic information. These key points include, but are not limited to, the corners of the eyes, pupils, nostrils, corners of the mouth, and jawline inflection points. Based on models such as convolutional neural networks, features are extracted layer by layer: low-level features: edges, textures, and brightness gradients, such as the contour of the corners of the eyes and the texture of the bridge of the nose; mid-level features: local semantic features, such as the combination of "double eyelids + high nose bridge"; high-level features: globally unique features, such as the combination of "jawline curvature + brow bone height." Among these, high-level features are the core for distinguishing different faces. Based on the detected bounding boxes and key points, the number of faces can be identified from an image frame. For example, 3 or 6 faces can be identified from a single image frame.

[0045] Furthermore, by analyzing single-frame images, if faces are detected in the image and the number of faces reaches or exceeds a preset face count threshold, the image is considered unsuitable for subsequent image selection and has little effect. Such images can be left unprocessed or edited to low priority. Image frames with a face count below the preset threshold are retained and marked as high priority. Finally, these high-priority image frames are used as keyframes. For example, if image frame S3 contains 8 faces and the preset face count threshold is 5, then image frame S3 is determined to be a complex extras scene and is skipped or marked as low priority. This process ensures that the selected keyframes focus on the main characters.

[0046] In this embodiment, by establishing keyframe filtering rules based on the number of faces, redundant frames with no valid faces or an excessive number of faces can be quickly filtered out, significantly reducing the amount of data processed for subsequent face feature extraction and identity recognition, and improving the overall efficiency of subsequent celebrity identity matching and role-real name association. At the same time, based on the statistics of face positions and numbers using bounding boxes and key points, and relying on mature face detection algorithms, a high counting accuracy can be guaranteed, reducing filtering deviations caused by false detections and missed detections.

[0047] In some optional embodiments, step S2022 above includes: Read a single video file and use a face detection algorithm to obtain the bounding box and key points in each image frame; The face location in each image frame is identified based on the bounding box and key points in each image frame; The face position is analyzed, and image frames with facial occlusion and / or image quality that does not meet the preset quality threshold are filtered out to retain the key frames.

[0048] In some examples, insufficient image clarity or facial occlusion can lead to face recognition failure. For instance, the face may be obscured by masks, sunglasses, hats, scarves, etc., obscuring key features such as the eyes, nose, and mouth. Other issues include low resolution / blurriness (images with pixels smaller than the standard 112×112 face size), excessive compression causing loss of facial details, direct sunlight causing overexposure, backlighting creating large shadows, excessive noise in low light, and excessive pose deviation (face angles exceeding the algorithm's tolerance range, such as a side yaw angle > 30°). To address these issues, after parsing a single video file and obtaining the bounding boxes and key points in each image frame using a face detection algorithm, the face location is locked. Face analysis can then be performed, filtering out frames with facial occlusion and / or image quality below a preset quality threshold, retaining the remaining frames as keyframes. Alternatively, keyframes can be further filtered based on the number of faces in the retained frames, ultimately selecting high-quality frames with fewer faces.

[0049] In this embodiment, by filtering image frames with facial occlusion and / or low image quality, high-quality keyframes are retained, which can improve the accuracy of face similarity calculation and thus more accurately merge faces of the same type.

[0050] In some optional embodiments, step S2023 above includes: For each episode of video, based on the face position extracted from the keyframe, the face features of each face are extracted using a preset face feature recognition model to construct the face feature vectors respectively, and the similarity between the face feature vectors is calculated. Based on the similarity between the facial feature vectors, faces in a single video set whose similarity exceeds a preset similarity threshold are classified to obtain multiple single video face sets. From each of the single video face sets, select the maximum facial region image of each type of face and the original image of the maximum single face.

[0051] In this embodiment, the aforementioned preset face feature recognition models include, but are not limited to, the ArcFace algorithm (Additive Angular Margin Loss), the CosFace algorithm (Large Margin Cosine Loss), and the SphereFace algorithm (Deep Hypersphere Embedding for Face Recognition).

[0052] In some examples, after selecting keyframes from a single video episode, face locations are located within these keyframes. The ArcFace algorithm is then used to extract facial features from these locations, constructing a facial feature vector (Embedding) for each detected face. This process involves: inputting the preprocessed face image into the ArcFace model; extracting features layer by layer from low-level texture to high-level semantics through convolution, pooling, and residual connections in the backbone network; and generating a fixed-dimensional original face feature vector through the model's feature output layer (512 dimensions, i.e., outputting a one-dimensional numerical array of length 512). Further L2 normalization is performed on the feature vectors, calculating the vector's magnitude and scaling it to 1. After normalization, the angle between the feature vectors represents the face similarity, eliminating the influence of feature magnitude on subsequent similarity calculations. The normalized 512-dimensional vector is the final constructed face feature vector, which can be used for subsequent face comparison, identity recognition and other tasks, such as calculating the cosine similarity of two face feature vectors to determine whether they are the same person.

[0053] Furthermore, based on the constructed facial feature vectors, the facial feature vectors of all faces detected in a single video episode are compared for similarity, and the cosine similarity of each facial feature vector within the single video episode is calculated. By setting a similarity threshold, the calculated similarities are compared with the threshold, and faces exceeding the similarity threshold are classified into the same category. For example, if face a and face b have a similarity of 85% and the similarity threshold is 80%, then face a and face b are classified into the same category. The classified faces of the same category in a single video episode are then considered as a single video face set, ultimately resulting in a set of single video faces within the single video episode.

[0054] Furthermore, all faces in the single-video face set for each face category can be compared, and two types of images can be selected for each face category: the largest facial region image and the largest single-face original image. Specifically, this can include: for the largest facial region image, calculating the face area of ​​all faces in the single-video face set based on the bounding boxes of the face regions, sorting them in descending order based on the face areas, and selecting the largest facial region image (masking) that ranks first. For the largest single-face original image, firstly, the original image containing only a single face is selected, then the face area is calculated based on the face region of the original image of the single face, sorted in descending order based on the face areas, and the image with the largest single-face facial region that ranks first is selected, i.e., the largest single-face original image.

[0055] In this embodiment, through a full-link design encompassing facial feature vector extraction, similarity comparison, clustering, and high-quality sample selection, optimization from video face detection results to high-precision recognition samples is achieved. This effectively solves the clustering problem of faces with the same identity in different image frames, laying a foundation for identity unification in subsequent sample selection. Based on the categorized single-video face set, the largest facial region image and the largest single face original image are selected, utilizing richer contextual information. This allows for the priority retention of high-quality face samples with the richest details and least interference, significantly improving the matching accuracy of celebrity face recognition services. Clustering eliminates the risk of cross-identity confusion, and area screening filters out low-quality, small-sized faces, significantly reducing the loss of recognition efficiency due to invalid samples.

[0056] In some optional embodiments, step S204 above includes: S2041, By comparing the facial regions of each type of face selected from all single episode videos with the original image of the largest single face, the global clustering results are obtained by globally aggregating them to the same global ID. The global clustering results include the global ID corresponding to each type of face. S2042, based on preset filtering conditions, select the global target face cutout image and the global target maximum single face original image from the global ID corresponding to each type of face.

[0057] In this embodiment, the aforementioned similarity algorithm can be a cosine similarity algorithm. After selecting the maximum facial region image and the maximum single face original image for each type of face in a single episode video, the faces selected from all single episode videos are compared again based on the facial feature vectors of the maximum facial region image and the maximum single face original image, and the same character appearing in different videos is grouped under the same global ID.

[0058] Furthermore, the aforementioned preset filtering conditions may include face area and number of images. For each global ID, from all the images aggregated, the global target face-cutting images with the highest face area ranking and matching the number of images are finally filtered out, along with the global target single-face original image with the largest face. For example, faces from 30 episodes of a single video are filtered out and clustered into the same global ID for comparison. The global target face-cutting images with the highest face area and no more than 5 images are selected, along with the global target single-face original image with the largest face, to serve as the "identity profile" for that character.

[0059] In this embodiment, a two-level cascaded clustering optimization architecture (video-level - album-level) is used to sequentially cluster similar faces and filter high-quality images as the data foundation for final identity recognition. This avoids the computational explosion problem caused by directly performing global clustering on massive video frames, greatly reducing the data dimensionality of album-level processing and improving processing speed. Image filtering based on preset selection criteria can extract more representative images from the largest facial region image and the largest single face original image, which will improve the accuracy and recall of subsequent external celebrity face recognition services.

[0060] In some optional embodiments, step S206 above includes: S2061, Based on the data input requirements of the external celebrity face recognition service, the format of the global target face cutout image and / or the original image of the largest single face of the global target is converted; S2062, based on the global target face-out large image after format conversion and / or the global target maximum single face original image, initiate an interface call request through a preset communication protocol to request the external celebrity face recognition service for identity matching; S2063, Obtain the return result based on the external celebrity face recognition service, the return result including the role information, the real information, and the mapping relationship between the role information and the real information.

[0061] For image format conversion, it is necessary to adapt to the data input requirements of the external celebrity face recognition service. The input object is at least one of the global target face-cutting large image and the global target largest single face original image. The selection of two types of images can improve the integrity of facial features and the recognition success rate. Format conversion is performed based on the data input requirements of the external celebrity face recognition service. The external celebrity face recognition service usually has explicit constraints on the input image, including image format, such as JPG / PNG / BMP, resolution, pixel depth, file size, color space, etc., to eliminate the input compatibility differences between the local image and the external service, ensuring that the image data can be correctly parsed and processed by the external service, and avoiding service call failure or recognition result distortion due to format incompatibility.

[0062] In some examples, the original image of the largest single face in the global target can be used first, as it contains more non-facial features, which helps improve recognition robustness. If the original image of the largest single face in the global target is not available, a large image of the face cut out from the global target can be used. Of course, both can be used together to improve accuracy and hit rate.

[0063] Furthermore, based on a pre-defined communication protocol, an interface call request is initiated to achieve cross-system communication and service invocation. The interface call request is initiated through the pre-defined communication protocol to invoke an external celebrity facial recognition service to complete identity matching. The pre-defined communication protocol includes, but is not limited to, standardized network communication protocols such as HTTP / HTTPS and gRPC. The interface call must comply with the request specifications defined by the external service, including request header parameters such as authentication tokens, data type identifiers, and request body structure. Using standardized protocols to achieve cross-platform and cross-system data transmission and service command distribution ensures the standardization and stability of the invocation process.

[0064] Furthermore, after completing internal processing steps such as image feature extraction, database comparison, and identity matching, the external celebrity facial recognition service returns response data, i.e., the results. These results include, but are not limited to, three types of key information: role information (such as the celebrity's character name in the video), real information (such as the celebrity's name), and the mapping relationship between role information and real information (i.e., establishing a one-to-one correspondence between "character in the show" and "real celebrity"). The returned results will be assigned to a corresponding global ID, completing the construction of the facial index for the entire series.

[0065] In this embodiment, targeted image format conversion solves the input compatibility problem between local images and external services, significantly reducing the failure rate of interface calls and improving the stability of service integration. Requests are initiated based on a preset protocol, ensuring the security and efficiency of cross-system data transmission and avoiding problems such as data loss, excessive latency, or parsing errors caused by protocol inconsistencies. Using a global target face-cutting image and the largest single-face original image as input ensures the integrity of facial features. Combined with the capabilities of external professional face recognition services, the accuracy of identity matching is improved. The returned results map role information to real information. Automated role construction for content not yet online fills the gap in character recognition for new dramas without internet metadata support, eliminating the need for manual pre-annotation.

[0066] It's important to understand that the information association method for unreleased videos provided in this application can be used in automated multimodal dataset production scenarios. It can automatically generate high-quality "face-character-dialogue" correspondence data for massive amounts of film and television dramas, directly empowering the training of large-scale multimodal models and significantly reducing the cost and time required for manual labeling. It can also be used for intelligent media asset management and search: for video platforms, structured processing can be completed before the video is released online. Users can directly locate specific segments in the drama by searching for "actor name" or "character name," improving user experience, such as the "only watch this character" function. Furthermore, it is applicable to video content understanding and review: assisting reviewers in quickly identifying sensitive figures or specific artists appearing in videos without requiring manual viewing of the entire video, thus improving review efficiency.

[0067] Combination Figure 4 As shown, to better understand the information association method for unreleased videos adopted in this application, a specific example will be used for illustration below.

[0068] First, at the video level, based on SCRFD face detection and ArcFace face recognition, SCRFD is used to obtain bounding boxes (bboxes) and keypoints (kpss), analyzing images containing up to five faces to find more facial images for each character. ArcFace face recognition is then used to calculate the embedding similarity of each face image, classifying faces exceeding a similarity threshold. Finally, a maximum facial region image and a maximum single-face original image are retained for each character. Further, at the album level, each video within an album is grouped together based on embedding similarity comparisons, resulting in a maximum of five global target face-outlining images and five global target maximum single-face original images for each character. Using the celebrity face recognition technology provided by Xingluo, the global target maximum single-face original image is prioritized for calling the celebrity face recognition interface; if no global target maximum single-face original image is available, the global target face-outlining image is used. Finally, each character is assigned the celebrity's real name and the character's name in the drama.

[0069] According to another aspect of the embodiments of this application, such as Figure 5 As shown, corresponding to the information association method for unreleased videos in the above embodiments, this embodiment provides an information association device for unreleased videos, the device comprising: The image acquisition module 501 is used to preprocess a single video to obtain the maximum facial region image and the maximum single face original image of each type of face in the key frame. The face clustering module 503 is used to perform global clustering based on the maximum facial region image and the original image of the maximum single face extracted from all single-episode videos for each type of face, and to select the global target face cutout image and the global target maximum single face original image for each type of face from the global clustering results. The identity recognition module 505 is used to call an external celebrity face recognition service to perform identity recognition based on the global target face cutout large image and / or the global target maximum single face original image, and associate each recognition object with real information and role information.

[0070] It should be noted that in this embodiment, the image acquisition module 501 can be used to execute step S202 in this application embodiment, the face clustering module 503 in this embodiment can be used to execute step S204 in this application embodiment, and the identity recognition module 505 in this embodiment can be used to execute step S206 in this application embodiment.

[0071] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should also be noted that the above modules, as part of a device, can operate in environments such as... Figure 1 The hardware environment shown can be implemented either through software or through hardware.

[0072] Optionally, the image acquisition module 501 specifically includes: a decomposition submodule, used to decompose the long video to be identified into multiple single-episode videos; a keyframe filtering submodule, used to perform face detection and feature extraction on each single-episode video, and filter the keyframes based on the extracted face features; and an image selection submodule, used to construct a face feature vector based on the face features in the keyframes of the single-episode video, and select the maximum facial region image and the maximum single face original image for each type of face based on the face feature vector.

[0073] Optionally, the keyframe filtering submodule includes: a first detection subunit, used to read a single-episode video file and obtain bounding boxes and key points in each image frame through a face detection algorithm; a face count subunit, used to determine the face positions in each image frame based on the bounding boxes and key points, and obtain the face count; and a comparison subunit, used to filter the keyframes according to the face count in each image frame, and mark image frames that include faces and whose face count is lower than a preset face count threshold as the keyframes.

[0074] Optionally, the keyframe filtering submodule includes: a first detection subunit, used to read a single-episode video file and obtain bounding boxes and key points in each image frame through a face detection algorithm; a face location detection subunit, used to identify the face location in each image frame based on the bounding boxes and key points in each image frame; and a filtering subunit, used to analyze the face location and filter image frames with facial occlusion and / or image quality that does not meet a preset quality threshold, so as to retain the obtained keyframes.

[0075] Optionally, the image selection submodule includes: a vector construction subunit, used to extract the facial features of each face in each single video episode based on the face positions extracted from the keyframes, using a preset facial feature recognition model to construct the facial feature vectors respectively, and calculate the similarity between the facial feature vectors; a face classification subunit, used to classify faces in the single video episode whose similarity exceeds a preset similarity threshold based on the similarity between the facial feature vectors, to obtain multiple single video face sets; and an image selection subunit, used to select the largest facial region image and the largest single face original image for each type of face from each single video face set.

[0076] Optionally, the face clustering module 503 includes: a global comparison submodule, used to perform comparison using a similarity algorithm, globally aggregating the largest facial region image of each type of face selected from all single-episode videos with the original image of the largest single face to the same global ID, to obtain the global clustering result, the global clustering result including the global ID corresponding to each type of face; and a global image filtering submodule, used to select the global target face cutout image and the original image of the global target largest single face from the global ID corresponding to each type of face based on preset filtering conditions.

[0077] Optionally, the identity recognition module 505 includes: a format conversion submodule, used to convert the format of the global target face-cutting image and / or the global target maximum single face original image based on the data input requirements of the external celebrity face recognition service; an interface call submodule, used to initiate an interface call request through a preset communication protocol based on the format-converted global target face-cutting image and / or the global target maximum single face original image to request the external celebrity face recognition service for identity matching; and a result acquisition submodule, used to acquire the return result based on the external celebrity face recognition service, wherein the return result includes the role information, the real information, and the mapping relationship between the role information and the real information.

[0078] It should be noted that the suffixes such as module, submodule, and subunit used to represent elements in the above-described device are only for the purpose of illustrative purposes and have no specific meaning in themselves. Therefore, they can be used in combination.

[0079] According to another aspect of the embodiments of this application, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the information association method for offline videos in any of the above embodiments.

[0080] According to another aspect of the embodiments of this application, this application also provides an electronic device, such as... Figure 6 As shown, the device includes a memory 601, a processor 603, and a network interface 605. The memory 601 stores a computer program that can run on the processor 603. The memory 601 and the processor 603 communicate through the network interface 605 and a communication bus 607. When the electronic device is running, the processor 603 and the memory 601 communicate through the network interface 605. When the processor 603 executes the computer program, it implements the steps of the above-described method for associating information with offline videos.

[0081] The memory and processor in the aforementioned electronic device communicate with each other via a communication bus and a communication interface. The communication bus can be a peripheral component interconnect standard (PCI) bus or an extended industry standard structure (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc. The memory can include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor. The aforementioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0082] It is understood that the embodiments described herein can be implemented using hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits, digital signal processors, digital signal processing devices, microprocessors, and other electronic units or combinations thereof for performing the functions described herein. For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. Software code can be stored in memory and executed by a processor.

[0083] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented using electronic hardware, or a combination of computer software and electronic hardware. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0084] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be electrical, mechanical or other forms.

[0085] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, the functional units in the various embodiments of this application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0086] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially, or the parts that contribute to the prior art, or parts of the technical solutions, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0087] It should be noted that, in this document, relational terms such as first, second, etc., are used only to distinguish one entity or operation from another entity or operation. The terms include, encompass, or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0088] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for associating information about videos that are not yet online, characterized in that, The method includes: Preprocess the single-episode video to obtain the maximum facial region image and the maximum single face original image for each type of face in the keyframes; Global clustering is performed based on the maximum facial region image and the original image of the maximum single face extracted from all single-episode videos for each type of face. From the global clustering results, the global target face-out image and the original image of the global target maximum single face are selected. Based on the global target face cutout image and / or the global target maximum single face original image, an external celebrity face recognition service is invoked to perform identity recognition, and each recognized object is associated with real information and role information.

2. The method for associating information about unreleased videos according to claim 1, characterized in that, The preprocessing of a single video episode to obtain the maximum facial region image and the maximum original image of a single face for each type of face in the keyframes includes: The long video to be identified is broken down into multiple individual video segments; Face detection and feature extraction are performed on each individual video episode, and the key frames are selected based on the extracted face features; A facial feature vector is constructed based on the facial features in the keyframes of a single video episode, and the maximum facial region image and the maximum single face original image for each type of face are selected based on the facial feature vector.

3. The method for associating information about unreleased videos according to claim 2, characterized in that, The step of performing face detection and feature extraction on each individual video segment, and then filtering the keyframes based on the extracted face features, includes: Read a single video file and use a face detection algorithm to obtain the bounding box and key points in each image frame; The face positions in each image frame are determined based on the bounding box and the key points, and the number of faces is obtained. The keyframes are filtered based on the number of faces in each image frame, and image frames that include faces and whose number of faces is less than a preset face count threshold are marked as keyframes.

4. The method for associating information about unreleased videos according to claim 3, characterized in that, The step of performing face detection and feature extraction on each individual video segment, and then filtering the keyframes based on the extracted face features, includes: Read a single video file and use a face detection algorithm to obtain the bounding box and key points in each image frame; The face location in each image frame is identified based on the bounding box and key points in each image frame; The face position is analyzed, and image frames with facial occlusion and / or image quality that does not meet the preset quality threshold are filtered out to retain the key frames.

5. The method for associating information about unreleased videos according to claim 3, characterized in that, The step of constructing a facial feature vector based on the facial features in the keyframes of a single video episode, and selecting the maximum facial region image and the maximum single-face original image for each type of face based on the facial feature vector, includes: For each episode of video, based on the face position extracted from the keyframe, the face features of each face are extracted using a preset face feature recognition model to construct the face feature vectors respectively, and the similarity between the face feature vectors is calculated. Based on the similarity between the facial feature vectors, faces in a single video set whose similarity exceeds a preset similarity threshold are classified to obtain multiple single video face sets. From each of the single video face sets, select the maximum facial region image of each type of face and the original image of the maximum single face.

6. The method for associating information about unreleased videos according to claim 1, characterized in that, The process involves global clustering based on the maximum facial region image and the original image of the largest single face extracted from all single-episode videos for each face type. From the global clustering results, a global target face-out image and the original image of the largest single face for each face type are selected, including: By comparing the facial regions of each type of face selected from all single-episode videos with the original image of the largest single face, the global clustering results are obtained by globally aggregating them to the same global ID. The global clustering results include the global ID corresponding to each type of face. Based on preset filtering conditions, the global target face cutout image and the original image of the largest single face of the global target are selected from the global ID corresponding to each type of face.

7. The method for information association of unreleased videos according to claim 1, characterized in that, The process involves using the large-scale face-cutting image of the global target and / or the original image of the largest single face of the global target to call an external celebrity face recognition service for identity verification, and associating the real information and role information of each identified object, including: Based on the data input requirements of the external celebrity face recognition service, the format of the global target face cutout image and / or the original image of the largest single face of the global target is converted; Based on the global target face-out large image after format conversion and / or the global target maximum single face original image, an interface call request is initiated through a preset communication protocol to request the external celebrity face recognition service for identity matching; Obtain the return result based on the external celebrity face recognition service. The return result includes the role information, the real information, and the mapping relationship between the role information and the real information.

8. An information association device for videos not yet online, characterized in that, The device includes: The image acquisition module is used to preprocess a single episode of video and obtain the maximum facial region image and the maximum single face original image for each type of face in the key frames. The face clustering module is used to perform global clustering based on the maximum facial region image and the original image of the maximum single face extracted from all single-episode videos for each type of face, and to select the global target face cutout image and the global target maximum single face original image for each type of face from the global clustering results. The identity recognition module is used to call an external celebrity face recognition service to perform identity recognition based on the global target face cutout large image and / or the global target maximum single face original image, and associate each recognized object with real information and role information.

9. An electronic device, comprising: A processor, a memory, and a network interface, wherein the memory stores machine-readable instructions executable by the processor, characterized in that: when the electronic device is running, the processor communicates with the memory via the network interface, and the processor executes the machine-readable instructions to perform the steps of the information association method for offline videos as described in any one of claims 1 to 7.

10. A storage medium having processor-executable non-volatile program code, characterized in that, The program code causes the processor to execute the steps of the information association method for offline videos as described in any one of claims 1 to 7.

Images