A method for generating face IDs and reconstructing lips in multi-shot video.

Abstract

This application relates to the technical field of computer vision processing, and in particular to a method for generating face IDs and reconstructing lips in multi-shot videos. The method includes: acquiring a video file and segmenting it into several shot fragments; performing intra-shot offline association within each shot fragment to generate at least one face trajectory; performing quality control based on each face trajectory to generate a trajectory-level representation; performing feature similarity comparison based on the trajectory-level representation to generate candidate merging pairs, and fusing these candidate merging pairs through cross-shot gating to obtain a global identity identifier; generating structured output; acquiring an audio file and obtaining several speech fragments, and assigning these speech fragments to target identities corresponding to the global identity identifier; and generating lip reconstruction results based on the lip reconstruction conditions for each target identity and outputting the reconstructed video. This application improves the consistency of global personnel identities in multi-shot videos.

Description

Technical Field

[0001] This application relates to the technical field of computer vision processing, and in particular to a method for generating face IDs in multi-shot video and a method for reconstructing lips in video. Background Technology

[0002] Video lip reconstruction technology aims to reconstruct or synchronize the lip movements of a target person in a video based on the input speech, so that the synthesized result matches the speech content. In actual video re-creation scenarios, the input video often contains multiple shots and multiple characters, and there are situations such as frequent shot cuts, multiple people on the same screen, occlusion, changes in posture, changes in lighting, and targets entering or leaving the frame.

[0003] Current common practices typically involve performing face detection and target tracking within a single shot to generate a trajectory within that shot, or using cross-segment face re-identification technology to extract appearance feature vectors from the trajectory or face, and performing similarity matching on different segments to merge the trajectories.

[0004] However, during engineering implementation, the following problems are likely to occur: trajectory breaks at camera transitions leading to tracking interruptions; the same person being assigned different IDs in different shots, making it difficult to achieve consistency across the entire video; false detections causing short trajectory interference during merging; low-quality face samples leading to unstable similarity and resulting in false merging / splitting; and the output results lacking an auditable, error-correctable, and directly usable structured format for downstream applications, thus increasing the cost of manual correction. Summary of the Invention

[0005] To improve the consistency of global personnel identities in multi-shot scenarios, this application provides a method for generating face IDs in multi-shot videos and a method for reconstructing lip images in videos.

[0006] Firstly, this application provides a method for generating face IDs in multi-shot videos, employing the following technical solution:

[0007] A method for generating face IDs in multi-shot videos includes the following steps:

[0008] The video file is acquired and segmented to obtain several shot clips;

[0009] In each of the aforementioned shot segments, face detection and key point localization are performed frame by frame, and detection results are generated. Based on the detection results, in-shot offline association is performed to generate at least one face trajectory.

[0010] Quality control is performed based on the face trajectories to generate trajectory-level representations, wherein the quality control includes short trajectory elimination and / or quality score evaluation.

[0011] Based on the trajectory-level representation corresponding to each of the face trajectories, feature similarity comparison is performed to generate candidate merging pairs, and the candidate merging pairs are fused through one or more cross-camera gating to obtain a global identity identifier and the mapping relationship between it and each of the face trajectories;

[0012] A global trajectory table is generated based on the mapping relationship, and a structured output is generated in combination with the global identity identifier. The global trajectory table includes the face box coordinates corresponding to the trajectory-covered frames and the zero-set coordinates corresponding to the trajectory-uncovered frames.

[0013] In some embodiments, face detection and key point localization are performed frame-by-frame in each of the said shot segments to generate detection results. Based on the detection results, in-shot offline correlation is performed to generate at least one face trajectory, including the following steps:

[0014] For each frame, perform face detection to obtain face bounding boxes and face bounding box coordinates, and perform key point localization to obtain several key points;

[0015] Based on the detection results, a motion prediction and cost matching association strategy is used in each of the aforementioned shot segments to associate each of the aforementioned face bounding boxes into the aforementioned face trajectory frame by frame.

[0016] Motion prediction is characterized by predicting the coordinates and motion state of the face bounding box corresponding to the current frame based on the trajectory state of the previous frame to obtain candidate bounding boxes;

[0017] A cost matrix is ​​constructed based on the candidate box and the detection box corresponding to the current frame to calculate the cost value. The factors in the cost matrix include at least one of the following: intersection-union ratio, center point distance, scale change penalty, and appearance similarity.

[0018] The cost matrix is ​​subjected to a Hungarian algorithm or a greedy algorithm to obtain inter-frame correlations to generate the face trajectory.

[0019] In some embodiments, performing in-lens offline correlation based on the detection results to generate at least one face trajectory further includes the following steps:

[0020] The shot segments are associated in both forward and reverse chronological order to obtain forward and reverse association results, respectively.

[0021] The positive and negative association results are compared for consistency, and the bidirectional association trajectories that pass the consistency comparison are fused to complete the verification of the face trajectory.

[0022] In some embodiments, quality control is performed based on the respective face trajectories to generate trajectory-level representations, including the following steps:

[0023] The face trajectories with a consecutive effective frame count lower than a preset threshold are removed, wherein the preset threshold is dynamically configured based on fps data;

[0024] And / or,

[0025] For each local face trajectory, a quality score is calculated for each face instance within the trajectory. The quality score is obtained by weighted calculation of several quality factors, including sharpness, face bounding box area, key point validity, pose deviation, and degree of occlusion / truncation.

[0026] From each face trajectory, several frames are selected sequentially in descending order of quality score as a high-quality sample set, and feature vectors are extracted from the high-quality sample set to obtain trajectory-level representation.

[0027] In some embodiments, feature vectors are extracted from the high-quality sample set to obtain a trajectory-level representation, including the following steps:

[0028] Based on the facial trajectory, the trajectory length and trajectory state are obtained, and a generation method for the trajectory-level representation is selected according to the computational requirements, including:

[0029] The frame with the highest quality score is selected as the best face frame, and the feature vector of the best face frame is extracted to obtain the trajectory-level representation.

[0030] or,

[0031] Feature vectors of all frames in the high-quality sample set are extracted to calculate the mean vector, and the trajectory-level representation is obtained based on the mean vector.

[0032] In some embodiments, the candidate merging pairs are fused through one or more cross-camera gating mechanisms to obtain a global identity and a mapping relationship between the identity and each of the face trajectories, wherein the cross-camera gating includes:

[0033] Temporal gating: If two face trajectories in the candidate merging pair have overlapping frames on the timeline, then the candidate merging pair is prohibited from passing.

[0034] Spatial displacement gating: Calculate the normalized distance between the end position of the first trajectory and the start position of the second trajectory in the candidate merging pair. If the normalized distance is greater than a preset threshold, the candidate merging pair is prohibited from passing.

[0035] IoU Gating: Calculate the cross-union ratio (CIU) of the two face trajectories in the candidate merging pair. If the CIU is lower than a preset threshold, the candidate merging pair is prohibited from passing.

[0036] In some embodiments, a global trajectory table is generated based on the mapping relationship, and a structured output is generated in conjunction with the global identity identifier, including the following steps:

[0037] The global identity identifier corresponding to each local face trajectory is determined based on the mapping relationship;

[0038] For each of the global identifiers, create a trajectory sequence with a length equal to the total number of video frames;

[0039] Fill the face bounding box coordinates of all video frames corresponding to the face trajectories belonging to the global identity into the corresponding frame positions in the trajectory sequence;

[0040] For video frames where the global identity identifier does not appear, preset zero coordinates are filled into the corresponding frame positions in the trajectory sequence.

[0041] In some embodiments, the structured output further includes:

[0042] It includes the index corresponding to the best face frame, the coordinates of the face bounding box, the pose, and the optimal frame information of the quality score;

[0043] Includes basic video information such as fps, a list of starting frames corresponding to each of the aforementioned shot segments, width and height, and the total number of video frames;

[0044] It includes diagnostic information such as the excluded trajectory information and the reason for gating rejection.

[0045] The technical solutions provided by the embodiments of this application have the following technical effects:

[0046] By employing "shot segmentation-driven processing + in-shot offline association + cross-shot gated fusion," global GIDs can be generated more stably under multi-shot conditions, reducing the occurrence of the same character being split or different characters being mistakenly merged, thus improving cross-shot identity consistency. High-quality samples are selected through quality control and then used for trajectory representation and fusion, which can reduce false detections and misfusions caused by low-quality appearance features. Structured output facilitates direct consumption by downstream modules, quality auditing, and interactive error correction, reducing manual post-processing costs. The generated global identity identifier and trajectory output can be used as pre-input for tasks such as re-dubbing, lip reconstruction, face swapping / beautification, and subtitle binding.

[0047] Secondly, this application provides a video lip reconstruction method, which adopts the following technical solution:

[0048] A video lip reconstruction method, based on the global identity identifier obtained from the above method, specifically includes the following steps:

[0049] The audio file is acquired and the speaker is separated to obtain several speech segments. Under the constraints of visibility, consistency and conflict resolution, the several speech segments are assigned to the target identity corresponding to the global identity identifier.

[0050] For the target identity, lip reconstruction conditions are constructed within the corresponding face trajectory range, and lip reconstruction results are generated and reconstructed video is output based on the lip reconstruction conditions. The lip reconstruction conditions include audio features, lip region mask, and identity constraint features.

[0051] The output is a structured output containing several of the aforementioned speech segments.

[0052] In some of these embodiments, the lip reconstruction conditions are specifically:

[0053] Audio features: characterized as time-frequency features extracted from the speech segment;

[0054] Lip region mask: Obtain the binary mask corresponding to the lip region based on key point recognition or segmentation model;

[0055] Identity constraint features: The feature vector corresponding to the best face frame is used as the identity constraint, or the mean vector in a high-quality sample set is used as the identity constraint.

[0056] The technical solutions provided by the embodiments of this application have the following technical effects:

[0057] Speech segments are generated by speaker separation / clustering and assigned to global identity identifiers to reduce the risk of audio-visual mismatch in multi-person scenarios. Local reconstruction and post-fusion under lip mask constraints concentrate changes in the lip region, reducing damage to non-lip regions and improving temporal stability. Attached Figure Description

[0058] Figure 1 This is a schematic diagram illustrating the steps of a multi-shot video face ID generation method provided in this embodiment.

[0059] Figure 2 This is a schematic diagram illustrating quality control and gating constraints in an embodiment of this application.

[0060] Figure 3 This is a schematic diagram of the steps of the video lip reconstruction method provided in the embodiments of this application.

[0061] Figure 4 This is a logical schematic diagram of the video lip reconstruction method provided in the embodiments of this application. Detailed Implementation

[0062] To better understand the purpose, technical solutions, and advantages of this application, it has been described and illustrated below with reference to the accompanying drawings and embodiments. However, those skilled in the art should understand that this application can be implemented without these details. In some cases, to avoid obscuring various aspects of this application due to unnecessary description, well-known methods, processes, systems, components, and / or circuits already described at a higher level will not be elaborated upon. It will be apparent to those skilled in the art that various modifications can be made to the embodiments disclosed in this application, and the general principles defined in this application can be applied to other embodiments and application scenarios without departing from the principles and scope of this application. Therefore, this application is not limited to the illustrated embodiments, but conforms to the broadest scope consistent with the scope of protection claimed in this application.

[0063] It should be noted that the descriptions of these embodiments are for the purpose of aiding understanding the present invention, but do not constitute a limitation thereof. Furthermore, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0064] In the description of this application, "several" means one or more, "more than" means two or more, "greater than," "less than," and "exceeding" are understood to exclude the stated number, while "above," "below," and "within" are understood to include the stated number. The use of "first" and "second" in the description is merely for distinguishing technical features and should not be construed as indicating or implying relative importance, or implicitly indicating the number of indicated technical features, or implicitly indicating the order of the indicated technical features.

[0065] In the description of this application, the terms "one embodiment," "some embodiments," "illustrative embodiment," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples.

[0066] like Figure 1 As shown in the figure, this application discloses a method for generating face IDs in multi-shot videos, including the following steps:

[0067] S100: Acquire video files and perform shot segmentation to obtain several shot clips.

[0068] First, transition detection is performed on the input video file to determine the boundaries between shots, thus obtaining several shot segments. Shot boundary determination can be achieved using inter-frame difference analysis, transition detection networks, or other equivalent methods.

[0069] Taking inter-frame difference detection as an example, each frame in the video file is first converted into the HSV color space. Each channel is divided into multiple bins and a multi-dimensional global color histogram is formed. The histogram is then normalized, and the value of each bin is converted to the range of [0, 1].

[0070] Furthermore, MobileNetV3 or other equivalent methods are used to extract multi-dimensional global semantic features of frames for subsequent hard / soft cut secondary verification, avoiding misjudging different scenes with similar colors as the same shot.

[0071] The difference between adjacent frames is calculated using Bach distance. The greater the difference, the greater the difference between the two frames, and the more likely it is to be a shot boundary.

[0072] In this application, the video is divided into hard cut and soft cut when the video is segmented into shots. Hard cut refers to a sudden change in the picture across shots, such as no transition between the previous frame and the next frame, or an instantaneous switch. Soft cut refers to transitional scenes such as fade-in and fade-out, dissolve, wipe, slow pan, etc., where the difference in the picture is a continuous and gradual change, rather than an instantaneous change.

[0073] In the hard-cut scenario, the default engineering coefficient k is obtained, and the hard-cut threshold is calculated based on μ+k×σ, where μ represents the mean of the Bach distance of all adjacent frames within the sliding window, and σ represents the standard deviation of the Bach distance of all adjacent frames within the sliding window.

[0074] If the Barthel distance between frame t and frame t-1 is greater than the hard cut threshold, and the Barthel distances of the three consecutive frames before and after frame t are all lower than the Barthel distance of that frame, then if the above conditions are met, then frame t-1 is the end frame of the previous shot, and frame t is the start frame of the next shot.

[0075] In soft cut scenarios, if the Bach distance of multiple consecutive frames is greater than μ+σ and exhibits a single-peak trend of first continuously increasing and then continuously decreasing, it is determined to be a soft cut transition interval. Record the start frame, end frame, and core overlapping frame interval (frames where multiple shots coexist during the transition). In soft cut scenarios, the frame preceding the start frame of the soft cut transition interval is the end frame of the previous shot, and the frame following the end frame of the transition interval is the start frame of the next shot.

[0076] Based on the above processing or other equivalent processing, a video file is sliced ​​into multiple shot segments. These shot segments may have spatiotemporal correlations but are shot from different camera positions or angles, may have weak spatiotemporal correlations, or may have no spatiotemporal correlations at all. However, several frames within each shot segment have strong spatiotemporal correlations.

[0077] Furthermore, after obtaining the shot segments, a list of starting frames, keyFrames, is obtained and recorded. Using these segmented shots as the basic processing unit, subsequent trajectory association is only performed within consecutive frames, reducing the risk of association and identity drift from the source. The starting frame list keyFrames, for example, is expressed as [0, 19, 256], representing that the video counts from frame 0, and the starting frames for each shot segment are 0, 19, and 256, respectively.

[0078] Furthermore, in addition to inputting video files and performing video segmentation, you can also directly select to input a list of pre-segmented and edited shot start frames (keyFrames).

[0079] S200 performs face detection and key point localization frame by frame in each shot segment and generates detection results. Based on the detection results, it performs offline correlation within the shot to generate at least one face trajectory.

[0080] For each frame in each shot segment, face detection is performed to obtain the bounding box (bbox). The bbox represents the coordinates of the face bounding box, which is expressed in the form of top-left / bottom-right corner, as [ ]; Normalized bbox is [ ].

[0081] Simultaneously, keypoint localization is performed to obtain keypoints, such as the commonly used 5-point or 68-point localization. Keypoints are used for: filtering face instances with invalid keypoints; assisting in pose estimation and quality scoring; and optionally, improving the stability of ReID features through face alignment.

[0082] Within the same shot segment, the detection results frame by frame are associated with face trajectories. This application employs an offline association method combining motion prediction and cost matching, with optional bidirectional association fusion to suppress drift. The trajectory is represented as the association result of the same target in the temporal dimension, including start and end frames and frame-by-frame bounding box sequences.

[0083] S300 performs quality control based on each face trajectory to generate trajectory-level representations, where quality control includes short trajectory elimination and / or quality score evaluation.

[0084] To reduce the interference of false detections and low-quality samples on cross-shot fusion, each trajectory is further subjected to short trajectory elimination and / or quality scoring. Short trajectory elimination is used to remove invalid trajectories with too few effective face frames or a small proportion of effective frames in the total number of trajectory frames. Quality scoring is used to obtain high-confidence, high-quality face samples based on the quality of face instances in different frames for identity representation.

[0085] Trajectory-level representation is a feature vector extracted and converged from the trajectory, used for cross-camera identity matching.

[0086] S400 performs feature similarity comparison based on the trajectory-level representation corresponding to each face trajectory to generate candidate merging pairs, and fuses the candidate merging pairs through one or more cross-camera gating to obtain the global identity identifier and the mapping relationship between it and each face trajectory.

[0087] For trajectory-level representations, appearance vector similarity is calculated to generate candidate matching relationships, and one or more gating constraints are superimposed to filter out unreasonable merging. Cross-camera trajectories that meet the candidate matching relationships and pass the gating constraints are fused, and a global identity identifier gid is generated using disjoint-set data structure / clustering. The gid is then mapped and associated with specific trajectories to obtain the mapping relationship rep_map.

[0088] S500 generates a global trajectory table based on the mapping relationship and generates a structured output by combining the global identity identifier. The global trajectory table includes the face box coordinates corresponding to the trajectory-covered frames and the zero-set coordinates corresponding to the trajectory-uncovered frames.

[0089] The mapping relationship is transformed into a structured global trajectory table, and together with the previously determined global identity identifier, a structured output is generated, so that the final output can be directly used for downstream re-production and interactive error correction.

[0090] Through the above steps, using "shot segmentation-driven processing + in-shot offline association + cross-shot gated fusion," global GIDs can be generated more stably under multi-shot conditions, reducing the occurrence of the same character being split or different characters being mistakenly merged, and improving cross-shot identity consistency. By using quality control to screen high-quality samples for trajectory representation and fusion, false detections and misfusions caused by low-quality appearance features can be reduced. Through structured output, it is easy for downstream modules to directly consume, conduct quality audits, and perform interactive error correction, reducing manual post-processing costs. The generated global identity identifier and trajectory output can be used as pre-input for tasks such as re-dubbing, lip reconstruction, face swapping / beautification, and subtitle binding.

[0091] Currently used visual processing methods, such as MOT and Re-ID, also employ techniques such as face detection, key point extraction, and trajectory generation. However, these methods differ from the actual scenarios and technical objectives of this application.

[0092] MOT primarily addresses how to keep track of a target in continuous video without losing its ID. Its default premise is that the video object being processed is basically continuous in time. Although there are fluctuations in the movement, position, and appearance changes between consecutive frames, the overall process is continuous. Re-ID, on the other hand, focuses on determining whether people in multiple cameras are the same person across cameras and time periods. Its core is more about identity matching at the appearance level.

[0093] This application deals with multi-shot videos, meaning that multiple video segments are inherently fragmented. Many logics that were originally valid in continuous video no longer hold true. For example, temporal continuity weakens, spatial relationships are severed, backgrounds may suddenly shift from indoors to outdoors, and a person may appear as a frontal view in one shot and a profile view in the next. Therefore, this application does not simply "identify who is who across shots," but rather unifies the identity of the same person throughout the entire multi-shot video, reorganizes it based on trajectory data, and finally outputs results that can be directly consumed downstream.

[0094] In other embodiments, face detection and key point localization are performed frame-by-frame in each shot segment, and detection results are generated. Based on the detection results, in-shot offline correlation is performed to generate at least one face trajectory, including the following steps:

[0095] S210: Perform face detection on each frame to obtain face bounding boxes and their coordinates, and perform key point localization to obtain several key points.

[0096] For each frame of the video clip, pixel normalization is performed, and the image is standardized to fit the prediction model's input requirements. Using current mainstream face detection models, the preprocessed frame images are input, and the face detection model outputs the pixel coordinates (bbox) of each face and the detection confidence score.

[0097] By setting a pre-defined confidence threshold, faces with confidence levels below the threshold are removed, thus filtering out false detections such as background posters and blurry images.

[0098] Perform face landmark detection and simultaneously output 5 facial key points: center of left eye, center of right eye, tip of nose, left corner of mouth, and right corner of mouth; if higher accuracy pose estimation is required, it can be replaced with a 68-point key point model to output the full face contour and facial feature detail key points.

[0099] The detected facial landmarks are further used for landmark validity verification and optional face alignment.

[0100] The key point validity verification is characterized by the fact that if a detected face meets any of the following criteria, it will be directly determined as an invalid face and removed from the subsequent steps:

[0101] Key points are outside the face bounding box;

[0102] The number of key points detected was insufficient;

[0103] If the pixel distance between the key points corresponding to the two eyes is less than the threshold, it indicates that the face size is too small and lacks detail.

[0104] The face bounding box extends beyond the video frame boundary, resulting in an incomplete face due to excessive truncation area.

[0105] For face alignment, the pose offset of the face is calculated based on key points. Through affine transformation, the two eyes are adjusted to the same horizontal line, and the face is corrected to a standard frontal pose. After alignment, the extracted identity features are more stable, and the similarity of features in different poses is higher, which greatly reduces the subsequent recognition error rate.

[0106] After face bounding box detection is completed and the face bounding box is determined to meet the processing requirements based on facial key points, it is necessary to generate the local trajectory corresponding to each person object in each shot segment based on offline association.

[0107] Within the same shot clip, the images are continuous, and the position of the same person between adjacent frames rarely changes abruptly (except in cases of occlusion). The next step is to string together the faces in each frame in chronological order, connecting the faces of the same person into a line, ultimately generating a trajectory. For example, if the shot includes a host and a guest, two independent trajectories will be generated, each recording the position of the two individuals in each frame from the beginning to the end of the shot.

[0108] The core of target tracking is inter-frame correlation: determining which face in the previous frame is the same person as the face in the current frame. The core criterion is that for the same target, the position, size, and appearance will not change abruptly between adjacent frames. Offline correlation uses frame information from the entire lens to perform one-way or two-way correlation, significantly reducing the probability of ID switching.

[0109] S220 uses a motion prediction and cost matching association strategy in each shot segment based on the detection results to associate each face bounding box with a face trajectory frame by frame.

[0110] Specifically, offline association includes two core steps: motion prediction and cost matching.

[0111] Motion prediction refers to predicting where a face will appear in the next frame based on the motion trajectory of a face in the previous frame.

[0112] It defines a reasonable candidate range for inter-frame matching, eliminating the need for indiscriminate searching among all faces in the entire frame, and only searching for matching objects near the predicted position; it provides a positional benchmark for subsequent cost matching, and all matching difficulty calculations are based on this prediction result; when a face is briefly occluded / out of the frame, the predicted position is used to temporarily complete the trajectory to avoid direct track breakage, and it can be quickly resumed when the face reappears.

[0113] Cost matching calculates a matching difficulty score (cost) for the predicted trajectory of the previous frame and the detected face in the current frame. The lower the score, the higher the probability that the two are the same person. Finally, the optimal pairing that corresponds one-to-one and does not conflict is found for all trajectories and face bounding boxes.

[0114] The core function of cost matching is to solve the core problem of "which trajectory of the previous frame does the face in the current frame belong to", stringing together the scattered single-frame face detection boxes into continuous trajectories; using a global optimal algorithm, it ensures that there will be no chaotic situations such as "multiple trajectories competing for the same face" or "one face being matched with multiple trajectories", and ensures that the trajectory IDs are not duplicated or missed.

[0115] S230, Motion prediction is characterized by predicting the coordinates and motion state of the face box corresponding to the current frame based on the trajectory state of the previous frame to obtain candidate boxes.

[0116] For each existing trajectory, the position of the current frame is predicted based on the position changes of historical frames. The general Kalman filter algorithm is used. The Kalman filter will simultaneously predict the position of the face, running speed, and acceleration, and continuously correct the prediction values ​​based on the historical motion state. Even if the person is briefly occluded, a reasonable position can be predicted to avoid losing track of them.

[0117] Alternatively, a simple replacement can be made using a uniform velocity model, which directly calculates the displacement of the center point and the change in the size of the face box between the previous and the frame before that. By assuming that the current frame moves with the same displacement and change, the predicted box can be calculated directly.

[0118] S240, construct a cost matrix based on the candidate box and the detection box corresponding to the current frame to calculate the cost value.

[0119] The cost is calculated between the predicted trajectory and the detection box in the current frame. The cost can consist of IoU, center point distance, scale change penalty, and optional appearance similarity.

[0120] The total cost is calculated once for each "active / lost trajectory of the previous frame" and each "detection box of the current frame", forming a cost matrix of "number of trajectories × number of detection boxes", which includes multiple dimensions.

[0121] IoU cost: The intersection-union ratio between the predicted bounding box of the trajectory and the detected bounding box of the current frame. The higher the IoU, the lower the cost. Usually, 1-IoU is used as the IoU cost. When IoU=1, the cost is 0, which indicates complete overlap; when IoU=0, the cost is 1, which indicates no overlap.

[0122] Centering distance cost: Calculate the Euclidean distance between the center point of the predicted bounding box and the center point of the detected bounding box in the image (normalization can be performed to eliminate the impact of resolution); when the face is temporarily obscured and the detected bounding box is small, the IoU may be very low, but if the center point distance is very close, it can be compensated accordingly through this dimension to avoid misjudging "not the same person".

[0123] Scale change penalty cost: Calculate the proportional difference between the width and height of the "predicted box" and the "detected box". The greater the difference, the higher the penalty. This cost calculation is used to avoid mismatching the "large box of other faces that suddenly enter the frame" with the "small predicted box of the original face", because the size of the face box of the same person will not change abruptly within a frame.

[0124] Optional appearance similarity cost: If high accuracy is required, a lightweight appearance feature (such as a simple color histogram or a small model Re-ID feature) can be extracted from the "face in the previous frame trajectory" and the "face in the current frame detection box" to calculate the feature similarity; the higher the similarity, the lower the cost, and 1-similarity is usually used as the appearance cost.

[0125] The costs of the above-mentioned dimensions are summed according to the preset weights to obtain the final total cost. The weights are adjusted according to the actual scenario. For example, for interviews with fixed camera positions, the IoU weight can be set higher; for outdoor scenes with large camera shake, the center point distance weight will be set higher.

[0126] S250, perform the Hungarian algorithm or greedy algorithm on the cost matrix to obtain inter-frame correlation to generate face trajectories.

[0127] After constructing the cost matrix, an algorithm is needed to find a face bounding box with the lowest total value that does not exceed the threshold for each trajectory, while ensuring that "one face bounding box matches only one trajectory, and one trajectory matches only one bounding box".

[0128] The algorithm can be either the Hungarian algorithm or a greedy matching algorithm:

[0129] The Hungarian algorithm automatically finds a one-to-one correspondence where the sum of the total costs of all pairs is minimized based on the input cost matrix. For each pair of matching results, it checks whether the total cost exceeds a preset matching threshold. If the total cost is not greater than the threshold, the match is successful and the detection box is associated with this trajectory. If the total cost is greater than the threshold, the match fails and this trajectory is temporarily lost.

[0130] Greedy algorithms are designed for scenarios with high speed requirements, and their accuracy is slightly lower than that of the Hungarian algorithm. They sort all "trajectory-detection box" pairs in ascending order of total cost, and start matching from the pair with the lowest cost. If a trajectory and a detection box have not been matched before, they are bound together, and this process is repeated until there are no pairs that meet the threshold.

[0131] Furthermore, trajectory maintenance can be performed to handle "unmatched" cases and ensure trajectory continuity. After trajectory matching is completed, two "unmatched" situations may occur: some trajectories do not find face detection boxes, and some face detection boxes do not find trajectories.

[0132] Specifically, the first step is to process the successfully matched trajectories, update the face detection box of the current frame to the state of this trajectory (replacing the old position box), reset the missing count gap of this trajectory to 0, keep the trajectory state active, and update the motion prediction model with the detection box of the current frame to make the next prediction result more accurate.

[0133] For a trajectory that is "not matched", the loss count gap of this trajectory is incremented by one, and it is checked whether the gap exceeds the maximum number of lost frames. If it does not exceed the maximum number of lost frames, the trajectory status is changed to "lost" but it is not terminated. The motion prediction model is used to predict the position of the next frame, and the matching is attempted again in the next frame. If it exceeds the maximum number of lost frames, the trajectory status is changed to "terminated" and it no longer participates in the matching of subsequent frames. This trajectory ends.

[0134] For detection boxes that "failed to match", which generally correspond to newly appearing faces, the confidence of the face detection box is checked first. Only face detection boxes with a confidence greater than a preset threshold will be used to initialize a new trajectory. A new in-lens trajectory is assigned to the newly appearing face box, and the state of this new trajectory is initialized.

[0135] After processing the last frame of a shot, all surviving tracks are terminated, and the set of tracks in that shot is output. Each track contains a local ID, start and end frame numbers, frame-by-frame bounding box sequence, frame-by-frame confidence sequence, etc.

[0136] In other embodiments, performing in-lens offline correlation based on the detection results to generate at least one face trajectory further includes the following steps:

[0137] S260, perform forward chronological association and reverse chronological association on the shot segments respectively, and obtain forward association results and reverse association results respectively.

[0138] S270, perform consistency comparison on the forward and reverse association results, and fuse the bidirectional association trajectories that have passed the consistency comparison to complete the verification of the face trajectory.

[0139] In offline association, bidirectional association fusion addresses the issue of unidirectional tracking easily leading to following the wrong person (ID switch) due to occlusion or blurring. Reverse tracking provides another dimension of verification, and the intersection of the two is used to correct the wrong person.

[0140] Specifically, all detection boxes within the same shot segment are first organized by frame number to form a list of face detection boxes, which includes the frame number of each frame, the bounding box of all face detection boxes in each frame, confidence level, appearance features, etc.

[0141] Secondly, based on the motion prediction and cost matching process in the above steps, the face detection box list is run sequentially to output the complete positive trajectory.

[0142] All frame numbers in the face detection box list are reversed and rearranged. An initial reverse trajectory is generated based on the high-confidence detection box of the reversed 0th frame (i.e. the last frame of the original shot). The motion prediction + cost matching process is repeated, and the trajectory obtained by reverse association is restored according to the frame number order of the original shot.

[0143] Establish a bidirectional mapping between detection boxes and trajectories. Specifically, first label each detection box with both positive and negative labels. The positive label is used to indicate which trajectory the detection box belongs to in the positive trajectory set, and the negative label is used to indicate which trajectory the detection box belongs to in the negative trajectory set.

[0144] Traverse all detection boxes and find consecutive frame intervals. If, within a certain consecutive frame interval, the forward and reverse trajectories are bound to the exact same (or highly overlapping) sequence of detection boxes, then this segment is used as an absolutely reliable seed trajectory.

[0145] Specifically, for example, if all detection boxes have a positive label of 1 and a negative label of 2 in consecutive frames 1-10, then this segment is marked as a "consistent segment" and the positive label 1 and negative label 2 are bound together.

[0146] Assign a temporary global ID to each "consistent segment" and include all detection boxes in that segment under that global ID as the basis for subsequent expansion.

[0147] After generating the seed trajectory mentioned above, expand it forward and backward respectively to fill in the parts that are "one-way present, one-way absent" or "have minor conflicts";

[0148] Conflict-free extension:

[0149] For example, if seed track 1 covers frames 1-10, forward track 1 also covers frames 11-20, and reverse track 2 has no other conflicting tags in frames 11-20, then frames 11-20 are also included in seed track 1.

[0150] Conflict handling (correcting ID switch):

[0151] For example, during forward tracking, if forward trajectory 1 is lost after frame 10 and becomes forward trajectory 2 in frame 11, while during reverse tracking, reverse trajectory 2 covers the area up to frame 20, then using the "consistent segment" as the anchor point, the binding of reverse trajectory 2 in frames 11-20 is prioritized and included in seed trajectory 1, correcting the forward ID switch. In case of a conflict, the result with "higher detection box confidence" and "higher trajectory continuity" is prioritized.

[0152] When video frames are missing, such as when a segment of a frame is occluded and neither the forward nor backward direction is tracked, but the seed trajectories before and after it are both global trajectory 1, the segment is marked as a missing segment of seed trajectory 1 and is completed through motion prediction.

[0153] In other embodiments, quality control is performed based on individual face trajectories to generate trajectory-level representations, including the following steps:

[0154] S310, removes face trajectories whose consecutive effective frame count is lower than a preset threshold, where the preset threshold is dynamically configured based on fps data.

[0155] The purpose of this step is to remove trajectories that are too short, as these short trajectories may be false samples caused by false detections.

[0156] First, determine the effective frame count threshold T_len, which is configured according to the video fps. For example, take the number of frames corresponding to 0.3s (e.g., for a 25fps video, T_len=7 frames). The effective frame count is represented by the number of frames in the trajectory that contain real face detection boxes, rather than the difference between the start and end frames.

[0157] If the number of valid frames is less than T_len, the trajectory is directly discarded and does not proceed to the next process, thereby filtering out invalid trajectories such as false background detections and fleeting passersby.

[0158] and / or

[0159] S320 calculates the quality score of each face instance within the trajectory for each local face trajectory, where the quality score is obtained by weighting several quality factors.

[0160] For each initial local face trajectory or the local face trajectory retained after the aforementioned short trajectory removal, a quality score q is further calculated. The higher the score, the better the face quality. In this embodiment, the score is obtained by weighted summation of 5 dimensions (the content and number of dimensions can be adjusted based on actual conditions), and each dimension is normalized to between 0 and 1.

[0161] The scoring dimensions specifically include: clarity, face frame area, key point validity, pose deviation, and degree of occlusion / truncation.

[0162] Among them, clarity is represented by the variance calculated after performing a Laplacian transform on the face detection bounding box region. The larger the variance, the clearer the image. A normalized score is obtained based on the variance and a preset transformation ratio. The face bounding box area is represented by the face bounding box area divided by the total area of ​​the video frames. The corresponding normalized score is mapped based on the area ratio and a preset transformation ratio. Keypoint validity is represented by the normalized result of the number of detected keypoints divided by the total number of keypoints plus the mean confidence score of the keypoints. Pose deviation is represented by the face yaw angle and pitch angle calculated based on the keypoints. If the absolute values ​​are both less than 30 degrees, 1 point is awarded; if both are greater than 90 degrees, 0 points are awarded. An intermediate linear mapping is performed accordingly. Occlusion truncation degree is represented by 1 point for no truncation or occlusion. When truncation or occlusion exists, the larger the corresponding truncation / occlusion area, the lower the corresponding normalized score.

[0163] Finally, the final quality score is obtained by combining the preset weighting coefficients for different quality factors.

[0164] The weighting can be adjusted for different videos and different shooting conditions, and can even be adaptively changed based on the statistical characteristics of the video itself.

[0165] For example, if faces in the current video are generally small, the distinguishing effect of the quality factor related to the face bounding box area will be more obvious, and the corresponding weight should be appropriately increased; if the characters' postures change significantly in the clip, with many instances of side profiles, heads tilted down, or heads tilted up, the weight of the posture factor needs to be appropriately increased; if there are many instances of occlusion, cropping, or out-of-frame phenomena, the weight of the occlusion / truncation factor can be appropriately increased; if the overall resolution is not low, but motion blur is significant, the weights of quality factors related to sharpness and clarity need to be increased.

[0166] Therefore, in some embodiments, the target video can be statistically analyzed first, such as the average face frame size, pose deviation distribution, occlusion ratio, and sharpness distribution. Then, the weights of each quality factor can be increased or decreased based on these statistical results, so that the final quality score is more in line with the "high-quality sample screening" requirements of the current video scenario.

[0167] S330 selects several frames from each face trajectory in descending order of quality score as a high-quality sample set, and extracts feature vectors from the high-quality sample set to obtain trajectory-level representation.

[0168] After calculating the quality score corresponding to each face trajectory, all faces within the trajectory are sorted in descending order of quality score q, and the top K frames are selected as the high-quality sample set topK, where K is 5 by default.

[0169] A trajectory-level representation is generated based on a high-quality sample set. The trajectory-level representation is the digital identity fingerprint of the task corresponding to the trajectory. It is a fixed-length feature vector used for subsequent cross-camera identity matching.

[0170] In other embodiments, feature vectors are extracted from a high-quality sample set to obtain a trajectory-level representation, including the following steps:

[0171] S331: Obtain the trajectory length and trajectory status based on the face trajectory, and select a generation method for trajectory-level representation based on the computational requirements.

[0172] The methods for generating trajectory-level representations based on a high-quality sample set topK vary depending on the trajectory and computing power scenario.

[0173] Specifically, it includes:

[0174] S332, select the frame with the highest quality score as the best face frame, and extract the feature vector of the best face frame to obtain the trajectory-level representation.

[0175] The frame with the highest quality score in each high-quality sample set is determined as the best face frame for that trajectory, and its frame number, bounding box, quality score, pose angle, and other data are recorded for subsequent auditing and display.

[0176] The feature vector of the best face frame is directly taken as the trajectory-level representation, which requires less computation.

[0177] and / or

[0178] S333 extracts the feature vectors of all frames in the high-quality sample set to calculate the mean vector, and obtains the trajectory-level representation based on the mean vector.

[0179] Alternatively, for each face in the topK high-quality sample set, input to the face recognition model to output a 512-dimensional face feature vector, and calculate the weighted average feature vector based on the quality score q of each sample. Samples with higher quality scores have a greater impact on the final representation. The final trajectory-level representation obtained by the above method is then L2 normalized to scale the vector length to 1. When calculating cosine similarity subsequently, the vector dot product is directly taken.

[0180] Specifically,

[0181] best_face is more like picking out the "most representative and highest quality face frame" from the trajectory.

[0182] Its advantages are that it is intuitive and simple, and this frame itself is of the best quality, usually with better clarity, pose and occlusion, so it is reasonable to use it as a representative.

[0183] It is suitable for the following types of scenarios:

[0184] The trajectory itself is relatively short, or there are not many high-quality frames; the face state is relatively stable within the same trajectory; more attention is paid to simple implementation and low computational cost.

[0185] The topK mean approach is different; it doesn't seek the "best frame," but rather constructs a "more stable trajectory-level representation."

[0186] In other words, trajectory-level representation is often more concerned with robustness than just achieving the highest quality in a single frame.

[0187] It is applicable to the following situations:

[0188] The trajectory is relatively long, and there are certain appearance changes within the same trajectory (such as slight changes in posture, expression, and lighting); more attention is paid to the stability of cross-camera matching to reduce the randomness of single frames; although there is a best_face in the trajectory, other high-quality frames also provide supplementary information.

[0189] After obtaining the trajectory-level representation, the cross-shot fusion stage is entered. For all cross-shot trajectory pairs, the potential trajectories with the same identity are first covered by coarse screening based on similarity, and then by hierarchical gating constraints.

[0190] The main reasons for this application's merging of cross-shot trajectories based on similarity include: in the same scene, people with similar appearances, or low-quality faces that are side-view, occluded, or blurred, may have extremely high feature similarity. Relying solely on the threshold will mistakenly merge different people into the same ID, leading to ID drift. Furthermore, for the same person in different shot segments, changes in lighting, angle, posture, and state can significantly reduce feature similarity. If the similarity threshold is set high, the same person may be mistakenly split into multiple IDs, resulting in identity fragmentation. If the threshold is set low, it will amplify the problem of erroneous merging.

[0191] Therefore, this application eliminates incorrect matching pairs that are "highly similar but logically impossible to be the same person" by setting gating constraints, and finally generates legal and mergeable trajectory pairs, thus solving the two core problems of incorrect merging and ID breakage.

[0192] First, iterate through all valid trajectories and calculate the cosine similarity sim(a,b) of the trajectory-level representations of any two trajectories a and b. A preset similarity threshold is set. Only when sim(a,b) is not less than the similarity threshold will a candidate merging pair be generated and proceed to the next step of gating screening.

[0193] The hierarchical gating check performs gating checks on each candidate merging pair in descending order of priority. If any gating rule prohibiting merging is triggered, the candidate pair is directly eliminated and merging is not allowed. Only candidate pairs that pass all effective gating checks are determined to be valid merging pairs.

[0194] In other embodiments, candidate merging pairs from one or more cross-camera gating methods are fused to obtain a global identity identifier and its mapping relationship with each face trajectory, wherein the cross-camera gating includes:

[0195] The first gate is a time-series gate: if the two face trajectories in a candidate merge pair have overlapping frames on the timeline, then the candidate merge pair is prohibited from passing.

[0196] Extract the time intervals of the two trajectories, such as trajectory a with start and end frames [start_a, end_a] and trajectory b with start and end frames [start_b, end_b], and determine if there is any temporal overlap. If max(start_a, start_b) is less than min(end_a, end_b), it means that the two trajectories intersect in time, indicating a visibility conflict. In this case, merging the two trajectories into the same GID means that the same character corresponds to two simultaneously visible instances at the same time, thus it can be determined as a conflict, and merging is prohibited.

[0197] Spatial displacement gating: Calculate the normalized distance between the end position of the first trajectory and the beginning position of the second trajectory in the candidate merging pair. If the normalized distance is greater than a preset threshold, the candidate merging pair is prohibited from passing.

[0198] First, the temporal continuity is verified. Spatial displacement gating only applies to trajectories connecting consecutive shots; that is, the ending frame (end_a) of trajectory a must be the last frame of the preceding shot, and the starting frame (start_b) of trajectory b must be the first frame of the following shot. Spatial displacement gating will only be applied if the above spatial continuity requirements are met. In other words, spatial displacement gating can only be applied to candidate trajectories when consecutive shots have visual continuity, scene continuity, or comparable composition.

[0199] Obtain the normalized center point coordinates of the face in the last frame before the end of trajectory a and the normalized center point coordinates of the face in the first frame of the beginning of trajectory b. Calculate the normalized Euclidean distance, where the value range of the Euclidean distance is [0, ...]. (The maximum distance corresponding to the diagonal of the screen). A preset displacement threshold T_d is used, and merging is directly prohibited when the normalized distance is greater than the displacement threshold T_d.

[0200] IoU Gating: Calculate the cross-union ratio (IoU) of the two face trajectories in the candidate merge pair. If the IoU is lower than a preset threshold, the candidate merge pair is prohibited from passing.

[0201] This gating is an optional supplementary check, and it is only performed on candidate pairs that meet the activation conditions. Specifically, IoU gating check requires that there are adjacent comparable frames or adjacent frames, and that the shot segments to which the two tracks belong are soft cut transitions.

[0202] Among them, adjacent comparable frames are characterized by two shots with overlapping frames, where the same frame contains the images from both shots and two face bounding boxes to be matched. The IoU of the two face bounding boxes within the same frame can be directly calculated. Adjacent frames are characterized by adjacent frames between shots in a slow cut / wipe transition, where the ending frame of the preceding shot is continuous with the starting frame of the following shot, and the face bounding box positions change continuously. The IoU of the face bounding boxes between the preceding and following frames can be calculated as a supplementary constraint. If there are relatively stable composition, scale, or viewpoint relationships between these frames, then IoU can be calculated as an auxiliary filtering method.

[0203] Soft cuts refer to transitions such as fade-in / fade-out, dissolve, wipe, and slow pan. These transitions often involve a transition process of several frames, with the image changing gradually rather than abruptly. These transitions are usually divided into two independent shots because the core subject / narrative unit changes, but there are overlapping / continuous frames between the two shots during the transition, which just meets the comparable frame prerequisite for IoU gating.

[0204] Specifically, the intersection-union ratio (CUI) of the bounding box (bbox) of the end frame of trajectory a and the bounding box of the start frame of trajectory b is calculated, and it is determined whether it is less than the preset CUI threshold. If it is, merging is directly prohibited.

[0205] The number and type of gating described above in this application will be selected based on the actual video segmentation feature scene. However, regardless of whether one or more gating are selected, if the condition corresponding to any single gating constraint is met, passage will be prohibited.

[0206] Some existing Re-ID technologies also perform some gating and filtering, such as combining time intervals, spatial changes, and even camera topology under different lenses to filter obviously unreasonable candidate matches.

[0207] However, the difference between this application and existing solutions of this kind lies not in the presence or absence of gating, but in the different scenarios and implementation logic upon which the gating is based. Traditional cross-shot Re-ID often assumes that there is still a strong spatiotemporal connection between different shots in the real world, and the time and space constraints are more about judging whether the migration of a person from shot A to shot B is reasonable; while we are dealing with multi-shot videos, where the shots may have been edited and spliced, and there may be significant jumps in background, framing, shooting angle and even temporal semantics between shots.

[0208] Therefore, the gating mechanism in this application is more suitable to be understood as a constrained fusion mechanism for multi-shot scenarios: it does not directly filter the original single-frame samples, but rather applies temporal, spatial, and IoU constraints to cross-shot trajectory candidate relationships after shot segmentation, intra-shot trajectory generation, quality control, and trajectory-level representation. The aim is to suppress unreasonable erroneous merging under weak or even discontinuous conditions, serving the subsequent global unification and structured output of face GID.

[0209] In other embodiments, a global trajectory table is generated based on the mapping relationship, and a structured output is generated by combining it with a global identity identifier, including the following steps:

[0210] S510 determines the global identity identifier corresponding to each local face trajectory based on the mapping relationship.

[0211] By merging all legal gating pairs, consistent trajectories are merged based on a disjoint-set data structure, generating a globally unique global identifier (gid).

[0212] Specifically, each valid trajectory is first assigned a unique initial ID. All valid merge pairs are iterated through, and the initial IDs corresponding to the two trajectories are merged into the same set in a union-find set. Each merged set corresponds to a unique global identity identifier (gid), and all trajectories within the set corresponding to each gid belong to the same person's facial identity. Simultaneously, each gid is renumbered sequentially according to the order in which it first appears in the video, facilitating downstream editing.

[0213] Disjoint set is the most efficient data structure for handling transitive merging, perfectly suited for engineering scenarios involving long videos and multiple tracks. At the same time, it can guarantee the absolute consistency of merging logic—for example, if tracks 1 and 2 can be merged, and tracks 2 and 3 can be merged, then 1, 2, and 3 will inevitably be grouped into the same set and assigned the same GID, completely avoiding the logical loophole of "the same person being split into multiple IDs," and fundamentally solving the problem of ID breakage across shots.

[0214] Meanwhile, the mapping table generated based on the mapping relationship between trajectory and corresponding GID records which global face ID corresponds to each trajectory within the shot.

[0215] S520 creates a trajectory sequence with a length equal to the total number of video frames for each global identity.

[0216] The trajectory sequence is generated based on the total number of frames of the entire video. It corresponds to a two-dimensional array, with each row corresponding to a gid. The number of rows corresponds to the number of different people, and each column corresponds to a frame of the video.

[0217] S530, fill the face bounding box coordinates of all face trajectories belonging to this global identity into the corresponding frame positions in the trajectory sequence.

[0218] S540, for video frames where the global identity does not appear, fill the corresponding frame position in the trajectory sequence with preset zero coordinates.

[0219] Iterate through each record in the mapping table, determine the global number of each trajectory based on the record, find all frames covered by the trajectory within the shot (e.g., from frame 10 to frame 50), find the row with the corresponding GID and the column of frames 10-50 in the trajectory sequence, and fill in the normalized face bounding box (bbox) of this trajectory in these frames.

[0220] Once all tracks corresponding to all GIDs have been filled in, the positions in the track sequence that have not been filled with any bboxes are filled with zero coordinates [0,0,0,0], indicating that no face frames appear in the video frames corresponding to these frames.

[0221] In other embodiments, the structured output also includes:

[0222] It contains the optimal frame information, including the index, face bounding box coordinates, pose, and quality score of the best face frame;

[0223] Basic video information including FPS, a list of starting frames for each shot, width and height, and the total number of video frames;

[0224] It includes diagnostic information such as the excluded trajectory information and the reason for gating rejection.

[0225] In addition to information such as the global identity identifier (gid) and the global trajectory table, the structured output also includes information related to the best face frame, basic video information, and control information corresponding to quality control.

[0226] Structured output is used to transform the results of all preceding steps into parameters that can be directly consumed later, such as for editing, auditing, and remake workflows. Together with shot segmentation, trajectory generation, and cross-shot fusion, it forms a complete closed loop.

[0227] This application also discloses a video lip reconstruction method, which is based on the global identity identifier obtained in the above method. A common use of the global identity identifier gid is to achieve controllable reconstruction and re-dubbing synthesis of lip regions with consistent identities in videos with multiple characters and multiple shots.

[0228] In actual video re-creation scenarios, the input video often contains multiple shots and multiple characters, and there are situations such as occlusion, changes in posture, changes in lighting, and objects entering or leaving the frame, which leads to:

[0229] The lack of a reliable mechanism for assigning voice clips to character identities, especially when multiple people are in the same scene or when shots are inserted across different scenes, can easily lead to the voice of one speaker being mistakenly assigned to the face of another person, resulting in lip mismatch.

[0230] Without constraints on the lip area or a local reconstruction strategy, unnecessary modifications can be made to non-lip areas, resulting in facial texture drift, flickering, or decreased stability.

[0231] This reduces the probability of identity shifts and lip mismatches when shooting across different shots.

[0232] Specifically, the following steps are included:

[0233] S600 acquires the audio file and performs speaker separation to obtain several speech segments, and assigns the speech segments to the target identities corresponding to the global identity identifiers under the constraints of visibility, consistency and conflict resolution.

[0234] In this application, the speaker is first separated or clustered on the input audio to obtain a set of speech segments, and the speech segments are then assigned to the target identity corresponding to the global identity identifier.

[0235] The input audio can be the original audio from the input video or external audio. When the input video does not contain available audio, a set of speech segments is generated using external audio and assigned, thereby achieving lip reconstruction through re-dubbing.

[0236] When the audio is the original audio of the video, speaker separation is performed. This is done by using a separation algorithm (such as Conv-TasNet) to separate the voices of different speakers in the mixed audio, assigning a unique speaker cluster ID to each independent speaker, and splitting the continuous speech of the same speaker into independent speech segments to form a complete set of speech segments.

[0237] When the audio consists of multiple external voiceovers, speaker clustering is performed. The clustering algorithm groups the segments of the same voiceover artist into one category and assigns them the same speaker cluster ID. If it is a single voiceover by a single person, it is directly assigned to a speaker cluster, and a corresponding set of audio segments is generated.

[0238] Each audio segment contains core fields: segment ID, start and end times, start and end frame numbers, speaker cluster ID, audio waveform, audio source, etc.

[0239] Furthermore, the speech segment allocation employs one or more combinations of the following constraints:

[0240] Visibility constraint: Allocation is only allowed during time periods when the facial trajectory of the target identity is visible and meets the quality standard. Specifically,

[0241] Extract the video frame interval corresponding to the audio segment, and check whether the face of the candidate GID is visible within the interval (i.e., the frame-by-frame bounding box is not [0,0,0,0]). Calculate the percentage of visible frames for the candidate GID within the interval. Only if the percentage reaches a preset threshold (generally ≥80% in engineering) and the average quality score of the visible frames meets the standard, is the allocation condition satisfied. If the GID is completely invisible within the frame interval of the audio segment, or the percentage of visible frames does not meet the standard, allocation is directly prohibited.

[0242] For example, if a certain audio segment corresponds to frames 100-200 of a video, and gid=1 is only visible for 10 frames in that range, while gid=2 is visible for 90 frames, then the segment will be preferentially assigned to gid=2.

[0243] Consistency constraint: This involves performing consensus voting or smoothing on the allocation results of the same speaker cluster over multiple time periods. Specifically,

[0244] For all speech segments of the same speaker cluster, the matching degree (visible frame ratio, quality score) of each candidate GID is calculated. A global voting mechanism is adopted, and the GID with the higher vote and the best matching degree is used as the unique target GID corresponding to the speaker cluster. All speech segments of the speaker cluster are assigned to the unique target GID, and the same sound segment is not allowed to be assigned to multiple people.

[0245] Conflict resolution: When multiple identities simultaneously meet the assignment criteria, a selection is made based on trajectory stability, occurrence duration, quality score, or coverage. Specifically,

[0246] When two or more GIDs simultaneously satisfy the visibility and consistency constraints mentioned above, the unique target GID is selected according to the following priority from high to low:

[0247] Trajectory coverage: Within the audio segment range, priority is given to the frames with the highest percentage of visible faces;

[0248] Face quality score: Prioritize the face with the highest average face quality score within the range;

[0249] Trajectory stability: Priority is given to tracks that are continuous and without frequent breaks throughout the entire video.

[0250] Total appearance time: Core characters with longer total appearance time throughout the entire video will be given priority.

[0251] To address extreme scenarios where multiple people are speaking simultaneously in the same frame, the system prioritizes assigning voice to key figures or clearly visible individuals in the frame, avoiding chaotic allocation.

[0252] S700 constructs lip reconstruction conditions within the corresponding facial trajectory range for the target identity, generates lip reconstruction results based on the lip reconstruction conditions, and outputs the reconstructed video.

[0253] This step is based on the voice segments allocated in S600. Within the trajectory range of the target GID, only the local area of ​​the lips is precisely reconstructed using audio-driven methods, while strictly keeping the non-lip areas completely unchanged. At the same time, identity constraints are used to ensure that the reconstructed face is consistent with the original person's identity.

[0254] Among them, the lip reconstruction conditions include audio features, lip region mask, and identity constraint features, which respectively solve the three core problems of audio-visual synchronization, region controllability, and identity consistency.

[0255] Specifically,

[0256] Audio features: These are represented as time-frequency features extracted from speech segments, specifically including:

[0257] Traditional video feature extraction is based on Mel spectrum and MFCC, adapted to lightweight generative models; deep speech features are extracted through deep semantic features from speech pre-trained models such as Hubert and Wav2Vec, which have higher accuracy in lip-syncing.

[0258] Perform temporal alignment, aligning the temporal resolution of the audio features with the video frame rate (e.g., for a 25fps video, each frame corresponds to 40ms of audio features), ensuring that the lip movements in each frame perfectly match the corresponding audio.

[0259] Audio features are the core input driving lip movements. The model generates corresponding lip opening and closing, movement patterns, and ensures that the mouth shape and the spoken content are completely synchronized.

[0260] Lip region mask: Obtain the binary mask corresponding to the lip region based on key point recognition or segmentation models. Specific methods include:

[0261] In keypoint recognition, several (e.g., 68) standard facial keypoints are detected for the target GID in the corresponding frame. Specific keypoints for the lip region (numbers 48-67, corresponding to the complete outline of the upper and lower lips) are extracted. Based on the lip outline points, a 10%-20% outward expansion is made to generate a closed binary mask region. The region inside the mask is marked as 1, which is the only lip region that can be reconstructed. The region outside the mask is marked as 0, which is a non-lip region that cannot be modified.

[0262] An insurmountable hard boundary was added to the reconstructed model. The model can only be modified within the local area of ​​the lips defined by the mask. It must never touch the non-lip areas such as the nose, cheeks, and eyes outside the mask, thus fundamentally preventing texture drift, flickering, and deformation in non-lip areas.

[0263] Identity constraint features: The feature vector corresponding to the best face frame is used as the identity constraint, or the mean vector in a high-quality sample set is used as the identity constraint.

[0264] From the full video trajectory of the target GID, select the top K high-quality sample set or directly use the best face frame corresponding to the GID. Through face recognition models such as ArcFace, extract the identity feature vectors of the selected high-quality faces as identity constraints in the reconstruction process.

[0265] This constraint addresses the common identity drift problem in audio-driven models by forcing the lip region generated by the model to perfectly match the original person's face identity. Even if the face is blurry or has a poor pose in a particular frame, the identity can still be guaranteed through high-quality global samples.

[0266] The steps for lip reconstruction are as follows:

[0267] Based on lip masking, a patch of the local lip region is extracted from the face in the original frame, and only this small region is input into the reconstruction model. Using audio features as motion effects and identity constraint features as identity anchors, the input lip patch is reconstructed to generate a new lip patch that is synchronized with the speech and consistent with the original person's identity.

[0268] Reconstruction or completion is performed within a local area defined by the lip mask. The reconstructed lip area is then merged and pasted back into the original frame, while non-lip areas remain unchanged or undergo weak fusion. Specifically...

[0269] The reconstructed lip patch is precisely pasted back to its corresponding position in the original frame, according to the original position of the mask, completely replacing the original lip area. Gaussian blur and feathering are applied to the edges of the mask to allow the reconstructed lip area to transition naturally with the original face skin area, avoiding obvious splicing marks and color block breaks. All areas outside the mask retain 100% of the original frame's image content without any modifications, completely avoiding image quality damage, distortion, and flickering in non-lip areas.

[0270] For consecutive reconstructed frames with the same GID, the motion trajectory of the lip region is smoothed between frames to avoid lip jumps and flickering, making the lip movements more natural and smooth. The temporal alignment between the reconstructed lip movements and the corresponding speech is checked frame by frame, and the deviation frames are fine-tuned to ensure that the mouth shape and the speech content are completely synchronized. All reconstructed frames are synthesized with the original audio / external dubbing audio, and the final lip reconstruction video is encoded and output, while retaining the resolution, frame rate and image quality of the original video.

[0271] The S800 outputs a structured output containing several speech segments.

[0272] The generated structured output will also include the start and end times of the speech segment set, speaker cluster identifiers, assigned GIDs, audio source markers, etc.

[0273] It should be understood that although the steps in the flowcharts in the accompanying drawings are shown sequentially as indicated by the arrows, these steps are not necessarily performed in the order indicated by the arrows. Unless otherwise expressly stated herein, there is no strict order in which these steps are performed, and they may be performed in other orders.

[0274] The above are all preferred embodiments of this application, and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.

Claims

1. A method for generating face IDs in multi-shot video, characterized in that, Includes the following steps: The video file is acquired and segmented into several shot segments. The shot segmentation method is based on inter-frame difference detection and is divided into hard cut and soft cut. Face detection and key point localization are performed frame-by-frame in each of the aforementioned shot segments, and detection results are generated. Based on the detection results, in-shot offline correlation is performed to generate at least one face trajectory. Specifically... The shot segments are associated in both forward and reverse chronological order to obtain forward and reverse association results, respectively. The forward and reverse association results are compared for consistency. The bidirectional association trajectories that pass the consistency comparison are then fused to complete the verification of the face trajectory. Specifically, Establish forward and reverse trajectory labels; use the sequence of detection boxes with consistent or highly overlapping forward and reverse labels within a continuous frame interval as seed trajectories; expand forward and backward based on the seed trajectories; correct the ID switch based on the detection box confidence and trajectory continuity when there is a conflict; Quality control is performed based on the aforementioned face trajectories to generate trajectory-level representations, wherein the quality control includes short trajectory removal and / or quality score evaluation, wherein the quality score evaluation includes: For each local face trajectory, a quality score is calculated for each face instance within the trajectory. The quality score is obtained by weighted calculation of several quality factors, including sharpness, face bounding box area, key point validity, pose deviation, and degree of occlusion / truncation. From each face trajectory, several frames are selected sequentially in descending order of quality score as a high-quality sample set, and feature vectors are extracted from the high-quality sample set to obtain trajectory-level representation. Based on the trajectory-level representations corresponding to each of the aforementioned face trajectories, feature similarity comparison is performed to generate candidate merging pairs. These candidate merging pairs, which are then fused through one or more cross-camera gating mechanisms, yield a global identity identifier and its mapping relationship with each of the aforementioned face trajectories. The cross-camera gating mechanism includes: Temporal gating: If two face trajectories in the candidate merging pair have overlapping frames on the timeline, then the candidate merging pair is prohibited from passing. Spatial displacement gating: Calculate the normalized distance between the end position of the first trajectory and the start position of the second trajectory in the candidate merging pair. If the normalized distance is greater than a preset threshold, the candidate merging pair is prohibited from passing. IoU Gating: Calculate the intersection-union ratio (IU) of comparable frames or adjacent frames of the two face trajectories in the candidate merging pair. If the IU is lower than a preset threshold, the candidate merging pair is prohibited from passing. A global trajectory table is generated based on the mapping relationship, and a structured output is generated in combination with the global identity identifier. The global trajectory table includes the face box coordinates corresponding to the trajectory-covered frames and the zero-set coordinates corresponding to the trajectory-uncovered frames.

2. The multi-shot video face ID generation method according to claim 1, characterized in that, Face detection and key point localization are performed frame by frame in each of the aforementioned shot segments, and detection results are generated. Based on the detection results, in-shot offline correlation is performed to generate at least one face trajectory, including the following steps: For each frame, perform face detection to obtain face bounding boxes and face bounding box coordinates, and perform key point localization to obtain several key points; Based on the detection results, a motion prediction and cost matching association strategy is used in each of the aforementioned shot segments to associate each of the aforementioned face bounding boxes into the aforementioned face trajectory frame by frame. Motion prediction is characterized by predicting the coordinates and motion state of the face bounding box corresponding to the current frame based on the trajectory state of the previous frame to obtain candidate bounding boxes; A cost matrix is ​​constructed based on the candidate box and the detection box corresponding to the current frame to calculate the cost value. The factors in the cost matrix include at least one of the following: intersection-union ratio, center point distance, scale change penalty, and appearance similarity. The cost matrix is ​​subjected to a Hungarian algorithm or a greedy algorithm to obtain inter-frame correlations to generate the face trajectory.

3. The multi-shot video face ID generation method according to claim 1, characterized in that, Quality control is performed based on the aforementioned face trajectories to generate trajectory-level representations, including the following steps: The face trajectories with a consecutive effective frame count lower than a preset threshold are removed, wherein the preset threshold is dynamically configured based on fps data.

4. The multi-shot video face ID generation method according to claim 3, characterized in that, Extracting feature vectors from the high-quality sample set to obtain trajectory-level representations includes the following steps: Based on the facial trajectory, the trajectory length and trajectory state are obtained, and a generation method for the trajectory-level representation is selected according to the computational requirements, including: The frame with the highest quality score is selected as the best face frame, and the feature vector of the best face frame is extracted to obtain the trajectory-level representation. or, Feature vectors of all frames in the high-quality sample set are extracted to calculate the mean vector, and the trajectory-level representation is obtained based on the mean vector.

5. The multi-shot video face ID generation method according to claim 1, characterized in that, A global trajectory table is generated based on the mapping relationship, and a structured output is generated by combining the global identity identifier, including the following steps: The global identity identifier corresponding to each local face trajectory is determined based on the mapping relationship; For each of the global identifiers, create a trajectory sequence with a length equal to the total number of video frames; Fill the face bounding box coordinates of all video frames corresponding to the face trajectories belonging to the global identity into the corresponding frame positions in the trajectory sequence; For video frames where the global identity identifier does not appear, preset zero coordinates are filled into the corresponding frame positions in the trajectory sequence.

6. The multi-shot video face ID generation method according to claim 5, characterized in that, The structured output also includes: It includes the index corresponding to the best face frame, the coordinates of the face bounding box, the pose, and the optimal frame information of the quality score; Includes basic video information such as fps, a list of starting frames corresponding to each of the aforementioned shot segments, width and height, and the total number of video frames; It includes diagnostic information such as the excluded trajectory information and the reason for gating rejection.

7. A video lip reconstruction method, characterized in that, The implementation is based on the global identity identifier obtained from the method described in any one of claims 1-6, and specifically includes the following steps: Acquire an audio file, and based on the source of the audio file, generate several speech segments by speaker separation, speaker clustering or single-person dubbing classification, and assign the several speech segments to the target identity corresponding to the global identity identifier under the constraints of visibility, consistency and conflict resolution. For the target identity, lip reconstruction conditions are constructed within the corresponding face trajectory range, and lip reconstruction results are generated and reconstructed video is output based on the lip reconstruction conditions. The lip reconstruction conditions include audio features, lip region mask, and identity constraint features. Based on the lip reconstruction conditions, under the constraints of the lip region mask, audio-driven reconstruction is performed on the local lip region in the frame corresponding to the target identity, while keeping the non-lip region unchanged. The output is a structured output containing several of the aforementioned speech segments.

8. The video lip reconstruction method according to claim 7, characterized in that, The specific conditions for lip reconstruction are as follows: Audio features: characterized as time-frequency features extracted from the speech segment; Lip region mask: Obtain the binary mask corresponding to the lip region based on key point recognition or segmentation model; Identity constraint features: The feature vector corresponding to the best face frame is used as the identity constraint, or the mean vector in a high-quality sample set is used as the identity constraint.

Images