Speaker localization and visual noise suppression method based on normalized lip movement analysis

Abstract

The present application belongs to the field of computer vision and multi-modal emotion analysis, and specifically relates to a speaker positioning and visual noise suppression method based on normalized lip movement analysis. In view of the fact that the existing multi-modal emotion analysis visual preprocessing lacks a systematic noise suppression process, and under the complex conditions of real video, multi-person co-occurrence, non-target speaker interference, profile occlusion and low light, the quality of the target speaker face sequence is difficult to be stably guaranteed, and the accuracy of subsequent visual feature extraction is affected, under the premise of not relying on data set pre-extraction features, the target speaker is automatically positioned from the original multi-person conversation video, and the face video segment with the target speaker face as the center and the output size of 224*224 is cropped, so as to systematically suppress multi-person co-occurrence, non-target speaker interference and profile occlusion from the data source level. The fixed threshold used is determined based on small-scale pre-experiments on training set and validation set samples, aiming to balance the detection recall rate, false detection suppression and trajectory stability.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and multimodal sentiment analysis, specifically a speaker localization and visual noise suppression method based on normalized lip movement analysis. Background Technology

[0002] Multimodal sentiment analysis is one of the core tasks in the fields of natural language processing and multimodal understanding, and it has significant application value in scenarios such as social media sentiment monitoring, intelligent human-computer interaction, and mental health assistance. With the rapid popularization of short video platforms, sentiment analysis methods that rely solely on text are no longer sufficient to cope with the complexity of emotional expression in real-world scenarios. Multimodal sentiment analysis that integrates non-verbal cues such as visual and audio data has become a mainstream research direction.

[0003] However, related studies often directly use pre-provided video clips or pre-extracted handcrafted visual features from datasets, treating preprocessing as a predetermined step. This lacks a systematic and specialized process for suppressing visual noise at the data source level, addressing complex situations commonly found in real videos, such as multiple speaker co-occurrence, interference from non-target speakers, side-face occlusion, and low lighting. Under these complex conditions, the quality of the target speaker's facial sequence is difficult to guarantee consistently, thus affecting the accuracy of subsequent visual feature extraction. Summary of the Invention

[0004] The purpose of this invention is to provide a speaker localization and visual noise suppression method based on normalized lip movement analysis. Starting directly from the original video, it can perform structured reconstruction of original videos containing interference from multiple people, occlusions, and non-target faces through speaker localization and face trajectory filtering. This reduces the impact of visual noise at the data processing source, providing a stable and reliable face video input for subsequent emotion feature extraction.

[0005] This method, without relying on pre-extracted features from the dataset, automatically locates the target speaker from the original multi-person conversation video and crops the output video segment to a size of 224×224, centered on the target speaker's face. It systematically suppresses visual noise such as multi-person co-occurrence, interference from non-target speakers, and side-face occlusion at the data source level. The fixed thresholds used in this invention are determined through small-scale pre-experiments using training and validation sets, aiming to balance detection recall, false positive suppression, and trajectory stability.

[0006] The method of this invention consists of seven stages, and includes a low-light recovery mode as a fallback, as detailed below: Phase 1: Multi-target face tracking Face detection was performed frame by frame in the video, with a detection confidence threshold set to 0.55. The output included face bounding boxes and two auxiliary keypoints: the tip of the nose and the center of the mouth. For each candidate box, a YCrCb skin color model was used to filter non-face regions: the region within the box was converted to the YCrCb color space, and the percentage of pixels falling within the skin color range (Cr∈[133,173], Cb∈[77,127]) was counted. Regions with a percentage of at least 10% and a width and height greater than 10 pixels were retained. For the retained face regions, an InceptionResNetV1 network pre-trained on the VGGFace2 dataset was used to extract 512-dimensional L2-normalized face embedding vectors.

[0007] To achieve cross-frame identity association, a cost matrix is ​​constructed between the existing trajectories in adjacent frames and the current detection, and the Hungarian algorithm is used to solve for the globally optimal binary matching. The cost function considers both spatial location consistency (IoU) and appearance similarity (cosine similarity), and is defined as: Where 0.4 and 0.6 are fixed thresholds, The bounding boxes of the faces in the two frames. This is the corresponding face embedding vector.

[0008] when When the cost of the pair is zeroed, it is assigned the highest priority in the Hungarian allocation; when merging similarity... The match will be accepted at that time.

[0009] Trajectories that are continuously lost for more than 15 frames are terminated and archived, while trajectories with an effective length of no less than 5 frames are retained as sub-trajectories.

[0010] Phase Two: Trajectory Merging and Candidate Selection When the same speaker is obscured or briefly leaves the frame, they may be segmented into multiple sub-trajectories during the tracking phase. The mean vector of all face embeddings for each sub-trajector is used as the identity feature of that trajectory.

[0011] A greedy strategy is used to merge sub-trajectories into existing identity clusters:

[0012] If the maximum cosine similarity between a sub-trajectory and an existing identity exceeds 0.68, it is merged into that identity; otherwise, it is registered as a new identity. After merging, linear interpolation is performed on the frame sequence of each identity (interpolation only occurs within consecutive frame intervals, with a maximum single interpolation span of 15 frames) to fill in temporary gaps. Finally, the identities are sorted in descending order of the number of frames they appear in, and the top 5 identities are selected as subsequent candidates.

[0013] Phase 3: Side Profile Candidate Filtering To reduce the interference of profile frames on subsequent lip movement analysis, candidate trajectories were sampled every 5 frames. MediaPipe FaceMesh was used to extract facial mesh key points, and the horizontal distance between the two corners of the mouth (index 61 and 291) was calculated as a surrogate indicator of the frontal appearance. , when The sampled frame is determined to be a frontal face frame. The frontal face rate is defined as the proportion of frontal face frames among the valid candidate sampled frames. Candidates with a frontal face rate below 35% are discarded; if no candidate meets this threshold, all candidates are retained as a fallback to prevent pipeline interruption.

[0014] Phase Four: Scene Type Determination When there are at least two valid candidates, select the two identities with the most occurrences in frames and calculate the overlap rate of their frame sequences: ,in, For identity The set of frame indices appearing in the video, molecule The number of frames in which both individuals appear simultaneously (i.e., the number of shared frames) is used. The denominator is the smaller of the number of frames in which both identities appear, to avoid underestimating the overlap rate due to one identity appearing for too short a time. When the overlap rate is no greater than 30%, it is determined to be a single-person cut scene, and active speaker identification is not performed subsequently. When the overlap rate is greater than 30%, it is determined to be a multi-person scene in the same frame, and proceeds to stage five. If there is only one valid candidate, it is directly processed as a single-candidate scene.

[0015] Phase 5: Normalized Lip Movement Analysis. For multi-person scenes, normalized lip movement analysis is performed frame-by-frame for each candidate, including two key processes: (1) Lip movement normalization. Divide the vertical distance between the key points of the upper and lower lips by the width of the corners of the mouth to obtain the dimensionless opening ratio: ;in These are the ordinates of key points on the upper and lower lips, respectively. These are the x-coordinates of the key points at the left and right corners of the mouth. To prevent smoothing terms resulting from division by zero, when the head pose changes, the width of the corners of the mouth and the distance between the lips change synchronously under the image projection. Dividing these two values ​​can partially offset the scale changes caused by the pose and eliminate the influence of individual differences in mouth shape.

[0016] (2) Pose-aware soft weighting. Frames are assigned continuous weights based on the visibility of the frontal face, replacing the hard drop-out strategy: Frontal face frame weight ; Slight side profile frame weighting Obvious side profile frame weights This is not included in the statistics.

[0017] (3) Calculate two statistics based on the weighted sequence: ; where the weighted mean Weighted standard deviation The opening state is changed from determination, This represents the cumulative number of times the open state has been flipped. The total weight is used to eliminate differences in video duration. CV (weighted coefficient of variation) describes the amplitude of lip movement, and DR (dynamic switching rate) describes the frequency of mouth opening state switching. Together, they characterize the motion pattern of speaking behavior.

[0018] Phase Six: Final Selection of the Three-Tier Speakers For scenarios where there are not multiple people in the same frame, the active speaker selection is not performed; instead, the candidate with the longest number of frames is directly selected as the target speaker.

[0019] For scenes with multiple people in the same frame, As the final speaking score, this product form penalizes two types of misjudgments simultaneously—occasional large mouth openings (high CV, low DR) and continuous slight jitter (high DR, low CV). A higher score is only generated when both the amplitude of the movement and the switching frequency are significant. A three-layer fallback mechanism was ultimately chosen: Tier 1: Candidates must simultaneously satisfy... and The speaker with the highest score is selected as the target speaker; Tier 2: If no candidate satisfies Tier 1, then... Take the score with the highest score. Tier 3: If there are no valid candidates in the first two tiers, then the candidate with the highest score is selected from all candidates.

[0020] This mechanism strictly identifies active speakers in scenarios with multiple people in the same frame, while ensuring that the pipeline can still produce output under extreme conditions.

[0021] Phase Seven: Stable Cropping and Video Output To output stable target speaker facial video, image stabilization is applied to the position and scale of the cropping window. First, shot transitions are detected using the mean absolute error (MAE) of adjacent grayscale frames downsampled to 1 / 4 resolution, with a threshold set to 28.0. When a shot transition, the first appearance of the target bounding box, a cropping scale change exceeding 30%, or a horizontal center offset exceeding 15% of the frame width is detected, the smoothing state is immediately reset to avoid cross-shot drift or abnormal smoothing.

[0022] Subsequently, an exponential moving average was applied simultaneously to the coordinates of the cutting center and the cutting side length. )smooth: , ,in, , The x and y coordinates of the crop center after smoothing in the current frame; , The smooth center of the previous frame; , The original center coordinates output by the detector; the historical weight of 0.9 makes the change in the position of the clipping box gradual, effectively suppressing detection noise.

[0023] The original cropping edge length is determined by multiplying the longer side of the target face bounding box by an expansion factor of 1.4 to preserve the appropriate facial context area.

[0024] in, , To detect the width and height of the bounding box and prevent the cropping window from exceeding the frame boundaries, a black-filled canvas is constructed around the original frame, and the target region is cropped from this canvas. The cropped result is then scaled to 224×224 and written to the output video. Black frames are inserted for frames where the target face is missing to maintain frame rate continuity and ensure strict time alignment between the output video and the original clip.

[0025] Low light recovery mode For samples where the mainstream waterline fails due to low light, a low-light recovery mode is activated. This mode lowers the face detection confidence threshold to 0.20, removes skin color filtering constraints, and relaxes the minimum trajectory survival length to 3 frames to capture weak face signals under extreme visual conditions; it also relaxes the upper limit of the interpolation span to 30 frames. After successful recovery, all detected candidate identities are smoothly cropped using EMA, and the target speaker is manually selected from the exported candidates; if a valid identity still cannot be detected, the sample is deemed invalid and discarded. Attached Figure Description

[0026] Figure 1 This is a flowchart illustrating the implementation technology of the present invention; Figure 2 This is a flowchart of Phase One of the present invention; Figure 3 This is a flowchart of stage two of the present invention; Figure 4 This is a flowchart of stage three of the present invention; Figure 5 This is a flowchart of stage four of the present invention; Figure 6 This is a flowchart of stage five of the present invention; Figure 7 This is a flowchart of stage six of the present invention; Figure 8 This is a flowchart of stage seven of the present invention; Figure 9 This is a flowchart of the low-light recovery mode of the present invention. Detailed Implementation

[0027] The technical solution of the present invention will be described in detail below with reference to specific embodiments and accompanying drawings. Those skilled in the art can reproduce the present invention based on these embodiments.

[0028] like Figure 1 As shown, a specific implementation of a robust speaker localization and visual noise suppression method based on normalized lip movement analysis according to an embodiment of this application is as follows: For the original conversation video containing the target speaker (which may contain single or multiple co-occurrence scenes, with no resolution limit), the target speaker is automatically located through the NAPS pipeline, and a face video segment with a size of 224×224 and centered on the target speaker's face is output.

[0029] In implementation, the first step is to prepare the runtime environment. This embodiment is based on Python and relies on open-source software libraries such as OpenCV, MediaPipe, facenet-pytorch, PyTorch, SciPy, and Pandas. No self-trained models are required; the face detector, face feature extraction network, and facial landmark detector all use publicly available pre-trained weights—the MediaPipe face detector and FaceMesh landmark detector are provided by MediaPipe, and the InceptionResNetV1 uses weights pre-trained on the VGGFace2 dataset. The running device can be a CPU or a CUDA GPU. The processing flow is executed sequentially in the following seven stages.

[0030] like Figure 2 As shown, Phase 1: Multi-target face tracking Face detection was performed frame by frame in the video, with a detection confidence threshold set to 0.55. The output included face bounding boxes and two auxiliary keypoints: the tip of the nose and the center of the mouth. For each candidate box, a YCrCb skin color model was used to filter non-face regions: the region within the box was converted to the YCrCb color space, and the percentage of pixels falling within the skin color range (Cr∈[133,173], Cb∈[77,127]) was counted. Regions with a percentage of at least 10% and a width and height greater than 10 pixels were retained. For the retained face regions, an InceptionResNetV1 network pre-trained on the VGGFace2 dataset was used to extract 512-dimensional L2-normalized face embedding vectors.

[0031] To achieve cross-frame identity association, a cost matrix is ​​constructed between the existing trajectories in adjacent frames and the current detection, and the Hungarian algorithm is used to solve for the globally optimal binary matching.

[0032] The cost function, which considers both spatial location consistency (IoU) and appearance similarity (cosine similarity), is defined as: ; where 0.4 and 0.6 are fixed thresholds, The bounding boxes of the faces in the two frames. This is the corresponding face embedding vector. When... When the cost of the pair is zeroed, it is assigned the highest priority in the Hungarian allocation; when merging similarity... The match is accepted. Trajectories with more than 15 consecutive lost detections are terminated and archived, while trajectories with a valid length of no less than 5 frames are retained as sub-trajectories.

[0033] like Figure 3 As shown, Phase Two: Trajectory Merging and Candidate Selection When the same speaker is obscured or briefly leaves the frame, they may be segmented into multiple sub-trajectories during the tracking phase. The mean vector of all face embeddings for each sub-trajector is used as the identity feature of that trajectory. A greedy strategy is used to merge sub-trajectories into existing identity clusters:

[0034] If the maximum cosine similarity between a sub-trajectory and an existing identity exceeds 0.68, it is merged into that identity; otherwise, it is registered as a new identity. After merging, linear interpolation is performed on the frame sequence of each identity (interpolation only occurs within consecutive frame intervals, with a maximum single interpolation span of 15 frames) to fill in temporary gaps. Finally, the identities are sorted in descending order of the number of frames they appear in, and the top 5 identities are selected as subsequent candidates.

[0035] like Figure 4 As shown, Stage 3: Profile Candidate Filtering To reduce the interference of profile frames on subsequent lip movement analysis, candidate trajectories were sampled every 5 frames. MediaPipe FaceMesh was used to extract facial mesh key points, and the horizontal distance between the two corners of the mouth (index 61 and 291) was calculated as a surrogate indicator of the frontal appearance.

[0036] when The sampled frame is determined to be a frontal face frame. The frontal face rate is defined as the proportion of frontal face frames among the valid candidate sampled frames. Candidates with a frontal face rate below 35% are discarded; if no candidate meets this threshold, all candidates are retained as a fallback to prevent pipeline interruption.

[0037] like Figure 5 As shown, Phase Four: Scene Type Determination When there are at least two valid candidates, select the two identities with the most occurrences in frames and calculate the overlap rate of their frame sequences: ,in, For identity The set of frame indices appearing in the video, molecule The number of frames in which both individuals appear simultaneously (i.e., the number of shared frames) is used. The denominator is the smaller of the number of frames in which both identities appear, to avoid underestimating the overlap rate due to one identity appearing for too short a time. When the overlap rate is no greater than 30%, it is determined to be a single-person cut scene, and active speaker identification is not performed subsequently. When the overlap rate is greater than 30%, it is determined to be a multi-person scene in the same frame, and proceeds to stage five. If there is only one valid candidate, it is directly processed as a single-candidate scene.

[0038] like Figure 6 As shown, Stage 5: Normalized lip movement analysis For scenarios with multiple people in the same frame, normalized lip movement analysis is performed frame-by-frame for each candidate, including two key processing steps: (1) Lip movement normalization: Divide the vertical distance between the key points of the upper and lower lips by the width of the corners of the mouth to obtain the dimensionless opening ratio: ,in, These are the ordinates of key points on the upper and lower lips, respectively. These are the x-coordinates of the key points at the left and right corners of the mouth. To prevent smoothing terms from being divided by zero.

[0039] When the head posture changes, the width of the corners of the mouth and the distance between the lips change synchronously under the image projection. Dividing the two can partially offset the scale changes caused by the posture and eliminate the influence of individual mouth shape differences.

[0040] (2) Pose-aware soft weighting. Frames are assigned continuous weights based on the visibility of the frontal face, replacing the hard drop-out strategy: The weight of the front view frame and the weight of the side view frame. Obvious side profile frame weights This is not included in the statistics.

[0041] (3) Calculate two statistics based on the weighted sequence:

[0042] Among them, the weighted mean Weighted standard deviation The opening state is changed from determination, This represents the cumulative number of times the open state has been flipped. The total weight is used to eliminate differences in video duration. CV (weighted coefficient of variation) describes the amplitude of lip movement, and DR (dynamic switching rate) describes the frequency of mouth opening state switching. Together, they characterize the motion pattern of speaking behavior.

[0043] like Figure 7 As shown, Stage Six: Final Selection of the Three-Tier Speakers For scenarios without multiple speakers in the same frame, no active speaker selection is performed; the candidate with the longest appearance count is directly selected as the target speaker. For scenarios with multiple speakers in the same frame, the active speaker is selected first. As the final speaking score, this product form penalizes two types of misjudgments simultaneously—occasional large mouth openings (high CV, low DR) and continuous slight jitter (high DR, low CV). A higher score is only generated when both the amplitude of the movement and the switching frequency are significant. A three-layer fallback mechanism was ultimately chosen: Tier 1: Candidates must simultaneously satisfy... and The speaker with the highest score is selected as the target speaker. Tier 2: If no candidate satisfies Tier 1, then... Take the score with the highest score. Tier 3: If there are no valid candidates in the first two tiers, then the candidate with the highest score is selected from all candidates.

[0044] This mechanism strictly identifies active speakers in scenarios with multiple people in the same frame, while ensuring that the pipeline can still produce output under extreme conditions.

[0045] like Figure 8 As shown, Stage Seven: Stable Cropping and Video Output To output stable target speaker facial video, image stabilization is applied to the position and scale of the cropping window. First, shot transitions are detected using the mean absolute error (MAE) of adjacent grayscale frames downsampled to 1 / 4 resolution, with a threshold set to 28.0. When a shot transition, the first appearance of the target bounding box, a cropping scale change exceeding 30%, or a horizontal center offset exceeding 15% of the frame width is detected, the smoothing state is immediately reset to avoid cross-shot drift or abnormal smoothing. Subsequently, an exponential moving average is applied simultaneously to the cropping center coordinates and the cropping side length. )smooth: , , in, , The x and y coordinates of the crop center after smoothing in the current frame; , The smooth center of the previous frame; , The original center coordinates output by the detector; the historical weight of 0.9 makes the change in the position of the clipping box gradual, effectively suppressing detection noise.

[0046] The original cropping edge length is determined by multiplying the longer side of the target face bounding box by an expansion factor of 1.4 to preserve the appropriate facial context area.

[0047] in, , To detect the width and height of the frame, and to prevent the cropping window from exceeding the frame boundary, a black-filled canvas is constructed around the original frame, and the target area is cropped from this canvas.

[0048] Finally, the cropped result is scaled to 224×224 and written to the output video. Black frames are inserted for frames where the target face is missing to maintain frame rate continuity and ensure that the output video is strictly time-aligned with the original clip.

[0049] like Figure 9 As shown, low light recovery mode For samples where the mainstream waterline failed due to low light, a low-light recovery mode is activated. This mode lowers the face detection confidence threshold to 0.20, removes skin color filtering constraints, and relaxes the minimum trajectory survival length to 3 frames to capture weak face signals under extreme visual conditions; it also relaxes the upper limit of the interpolation span to 30 frames. After successful recovery, all detected candidate identities are smoothly cropped using EMA, and the target speaker is manually selected from the exported candidates; if a valid identity still cannot be detected, the sample is deemed invalid and discarded.

[0050] Example 1 Group 1 (NAPS Overall Localization Accuracy): NAPS performed speaker localization on all 4403 samples, with an automatic localization accuracy of approximately 95% for the main line. Seven samples were discarded due to consistently side profiles / severe blurring / no valid faces, resulting in the effective cropping of 4396 samples. Group 2 (Core Innovation – Normalized Lip Movement + Pose Soft-Weighted Ablation): 100 multi-person co-occurrence candidate samples were labeled from CH-SIMS v2.0S. After filtering, 81 valid multi-person co-occurrence samples were retained. The speaker recognition accuracy was compared between Abs-Hard (old version) and Norm-Soft (this method) – Abs-Hard was 70 / 81 = 86.4%, while Norm-Soft was 73 / 81 = 90.1%, representing an improvement of 3.7 percentage points.

[0051] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this invention.

Claims

1. A speaker localization and visual noise suppression method based on normalized lip movement analysis, characterized in that, Includes the following steps: Phase 1: Multi-target face tracking Face detection is performed frame by frame in the video, with a detection confidence threshold of 0.

55. The bounding boxes of the faces and two auxiliary key points, the tip of the nose and the center of the mouth, are output. For each candidate box, the YCrCb skin color model is used to filter out non-face regions: the region inside the box is converted to the YCrCb color space, and the percentage of pixels falling within the skin color range Cr∈[133,173] and Cb∈[77,127] is counted. When the percentage is not less than 10% and the width and height of the box are both greater than 10 pixels, it is retained. For the retained face regions, the InceptionResNetV1 network pre-trained on the VGGFace2 dataset is used to extract 512-dimensional L2 normalized face embedding vectors. Phase Two: Trajectory Merging and Candidate Selection The same speaker may be segmented into multiple sub-trajectories during the tracking phase due to occlusion or brief absence from the frame. The mean vector of all face embeddings for each sub-trajector is used as the identity feature of that trajectory. ; Phase 3: Side Profile Candidate Filtering To reduce the interference of side profile frames on subsequent lip movement analysis, candidate trajectories are sampled every 5 frames. MediaPipeFaceMesh is used to extract facial mesh key points, and the horizontal distance between the two corners of the mouth key points is calculated as a proxy index for the frontal face. Phase Four: Scene Type Determination When there are at least two valid candidates, select the two identities with the most occurrences in frames and calculate the overlap rate of their frame sequences: ,in, For identity The set of frame indices appearing in the video, molecule The number of frames in which the two people appear on screen at the same time, i.e. the number of shared frames, is calculated by taking the smaller value of the number of frames in which the two identities appear, in order to avoid the overlap rate being underestimated due to one identity appearing for too short a time. When the overlap rate is no more than 30%, it is determined to be a single-person cut scene, and active speaker identification will not be performed subsequently; when the overlap rate is greater than 30%, it is determined to be a multi-person scene, and enters stage five. If there is only one valid candidate, it will be processed directly as a single candidate scene. Phase 5: Normalized Lip Movement Analysis. For multi-person scenes, normalized lip movement analysis is performed frame-by-frame for each candidate, including two key processes: (1) Lip movement normalization: divide the vertical distance between the key points of the upper and lower lips by the width of the corners of the mouth to obtain the dimensionless opening ratio. (2) Pose-aware soft weighting assigns continuous weights to frames based on the visibility of the frontal face, replacing the hard discard strategy; two statistics are calculated based on the above two weighted sequences; Phase Six: Final Selection of the Three-Tier Speakers For scenarios where there are not multiple people in the same frame, the active speaker selection is not performed; instead, the candidate with the longest number of frames is directly selected as the target speaker. Phase Seven: Stable Cropping and Video Output Output a stable video of the target speaker's face, apply image stabilization to the position and scale of the cropping window, and detect camera switching by the mean absolute error (MAE) of adjacent grayscale frames downsampled to 1 / 4 resolution, with a threshold set to 28.

0.

2. The speaker localization and visual noise suppression method based on normalized lip movement analysis according to claim 1, characterized in that, In Phase 1, the cross-frame identity association step involves constructing a cost matrix between the existing trajectories in adjacent frames and the current detection, and then using the Hungarian algorithm to solve for the globally optimal binary match. The cost function considers both spatial location consistency and appearance similarity, and is defined as follows: ； Where 0.4 and 0.6 are fixed thresholds, The bounding boxes of the faces in the two frames. For the corresponding face embedding vector; when When the cost of the pair is zeroed, it is assigned the highest priority in the Hungarian allocation; when merging similarity... The matching is accepted when the time is right. Trajectories that are continuously lost for more than 15 frames are terminated and archived. Trajectories with a valid length of no less than 5 frames are retained as sub-trajectories.

3. The speaker localization and visual noise suppression method based on normalized lip movement analysis according to claim 1, characterized in that, In Phase Two, a greedy strategy is used to merge the sub-trajectories into the existing identity cluster: ; If the maximum cosine similarity between a sub-trajectory and an existing identity exceeds 0.68, it is merged into that identity; otherwise, it is registered as a new identity. After merging, linear interpolation is performed on the frame sequence of each identity, interpolating only within consecutive frame intervals, with a maximum single interpolation span of 15 frames to fill in temporary gaps; finally, the identities are sorted in descending order of the number of frames they appear in, and the top 5 identities are selected as subsequent candidates.

4. The speaker localization and visual noise suppression method based on normalized lip movement analysis according to claim 1, characterized in that, In Phase 3, the key points at the corners of the mouth are defined as indices 61 and 291, and the absolute difference is taken. ;when The sampling frame is determined to be a frontal face frame. The frontal face rate is defined as the proportion of frontal face frames in the candidate valid sampling frames. Candidates with a frontal face rate of less than 35% are eliminated. If no candidate meets the threshold, all candidates are retained as a fallback to prevent pipeline interruption.

5. The speaker localization and visual noise suppression method based on normalized lip movement analysis according to claim 1, characterized in that, In phase five, The opening ratio is ,in These are the ordinates of key points on the upper and lower lips, respectively. These are the x-coordinates of the key points at the left and right corners of the mouth. To prevent smoothing terms from being divided by zero; When the head posture changes, the width of the corners of the mouth and the distance between the lips change synchronously under the image projection. Dividing the two can partially offset the scale changes caused by the posture and eliminate the influence of individual mouth shape differences.

6. The speaker localization and visual noise suppression method based on normalized lip movement analysis according to claim 1, characterized in that, In phase five, the discarding strategy is as follows: frontal face frame weight Slight side profile frame weight Obvious side profile frame weights This is not included in the statistics.

7. The speaker localization and visual noise suppression method based on normalized lip movement analysis according to claim 5 or 6, characterized in that, Calculate two statistics based on the weighted sequence: , Among them, the weighted mean Weighted standard deviation The opening state is changed from determination, This represents the cumulative number of times the open state has been flipped. The total weight is used to eliminate differences in video duration; the weighted coefficient of variation (CV) characterizes the amplitude of lip movements, and the dynamic switching rate (DR) characterizes the frequency of mouth opening state switching. Together, they represent the motion pattern of speaking behavior.

8. The speaker localization and visual noise suppression method based on normalized lip movement analysis according to claim 1, characterized in that, In phase six, For scenes with multiple people in the same frame, As the final speaking score; this product form can simultaneously penalize two types of misjudgments—occasional large mouth opening, high CV, low DR, and continuous slight shaking, high DR, low CV; only when both the amplitude of movement and the switching frequency are significant will a higher score be generated, and a three-layer fallback mechanism is ultimately chosen: First level: Requires candidates to simultaneously satisfy... and The speaker with the highest score is selected as the target speaker. Second level: If no candidate satisfies the first level, then it requires Take the score with the highest score. The third layer: If there are no valid candidates in the first two layers, the candidate with the highest score is selected from all candidates. This mechanism strictly identifies active speakers in multi-person scenarios while ensuring that the pipeline can still produce output under extreme conditions.

9. The speaker localization and visual noise suppression method based on normalized lip movement analysis according to claim 1, characterized in that, In phase seven, when a camera switch, the first appearance of the target frame, a cropping scale change of more than 30%, or a horizontal center offset of more than 15% of the frame width is detected, the smoothing state is immediately reset to avoid cross-camera drift or abnormal smoothing.

10. The speaker localization and visual noise suppression method based on normalized lip movement analysis according to claim 9, characterized in that, Apply an exponential moving average to both the cutting center coordinates and the cutting side length. smooth: , ,in, , The x and y coordinates of the crop center after smoothing in the current frame; , The smooth center of the previous frame; , ; The original center coordinates output by the detector; a history weight of 0.9 makes the change in the clipping box position gradual, effectively suppressing detection noise; The original cropping edge length is determined by multiplying the longer side of the target face bounding box by an expansion factor of 1.4 to preserve the appropriate facial context area. ,in, , To detect the width and height of the bounding box, and to prevent the cropping window from exceeding the frame boundary, a black-filled canvas is constructed around the original frame, and the target area is cropped from this canvas. Finally, the cropping result is scaled to 224×224 and written to the output video. Black frames are inserted for frames where the target face is missing to maintain frame rate continuity and ensure that the output video is strictly time-aligned with the original clip.

Images