A method for speaker recognition based on lip movement

Abstract

This invention relates to the field of speaker recognition, specifically to a speaker recognition method based on lip movements. The method includes: constructing an event-based lip movement dataset; constructing a lip movement-based speaker recognition model, the model comprising a preprocessing stage, performing event-level denoising to suppress spurious events, and applying robustness enhancement to mitigate the impact of scene changes and noise; a time-aware voxel encoding stage, mitigating the impact of quantization errors by converting sparse asynchronous events into dense and information-rich tensors; a structure-aware spatial enhancement stage, refining voxelized features through channel compression layers and orientation-aware spatial smoothing to reduce redundancy and enhance spatial representation; a classification and polarity consistency regularization processing stage; training the lip movement-based speaker recognition model using the event-based lip movement dataset; and performing speaker recognition after training. This invention is applicable to speaker recognition.

Description

Technical Field

[0001] This invention relates to the field of speaker recognition, and more specifically to a speaker recognition method based on lip movements. Background Technology

[0002] Speaker recognition technology identifies individual speakers by analyzing visual cues during speech. As an inherent biometric feature, lip morphology possesses stable and distinguishable characteristics, making it highly suitable for personal identification. Specifically, speech articulation involves the co-activation of complex lip anatomy and facial motor units, forming unique spatiotemporal patterns that encode each individual's unique identity characteristics. These patterns are difficult to imitate or synthesize realistically, thus exhibiting natural resistance to impersonation attacks. Based on its biological characteristics, non-invasive acquisition methods, and natural resistance to environmental acoustic noise, lip-based speaker recognition technology has become a promising and practical solution for secure access control in applications such as mobile banking identification, intelligent monitoring, and biometric forensics.

[0003] Lip features can generally be categorized into three types: lip texture, lip geometry, and lip movement. Lip texture and geometry are static physiological biometrics, describing the surface morphology and overall lip shape, respectively. Previous research has primarily focused on these static physiological signals, often relying on prior knowledge (such as Zernike transform-based geometric descriptors and sparse coding of local texture blocks) to model appearance features or texture-based identity features. In contrast, lip movement, as a behavioral biometric, characterizes the spatiotemporal dynamic changes of the lips and perioral muscles during pronunciation, and its use as an identity feature has been increasingly studied in recent years. Due to the high correlation between motion information and appearance features, existing methods (especially end-to-end methods based on deep learning) often fuse static and dynamic information in an entangled manner. This fusion method inevitably leads to the learned identity representation inheriting the sensitivity of appearance features to imaging conditions.

[0004] Therefore, existing speaker recognition technologies based on lip features are extremely sensitive to changes in acquisition conditions. In practical applications, changes in viewing angle and lighting conditions can lead to significant distribution shifts: the lip geometry may be deformed or partially occluded from a side view, while insufficient lighting can blur the subtle textures of the lips. This mismatch between training and testing data can cause a significant drop in recognition accuracy. Furthermore, in real-world scenarios, users typically provide limited training samples under controlled conditions, while testing often takes place in unconstrained environments. This raises a cross-scenario problem with significant practical value that has not yet been fully explored: how to learn lip feature-based identity representations from a single scenario during the training phase while ensuring robustness to unknown changes in viewing angle and lighting conditions during testing. This scenario is particularly important when repeatedly acquiring data under multiple conditions is too costly or difficult to operate, or when the potential deployment environment space cannot be exhaustively enumerated. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a speaker recognition method based on lip movements, which realizes cross-scene speaker recognition by utilizing fine dynamic lip movements.

[0006] The present invention achieves the above objectives by adopting the following technical solution: The present invention provides a speaker recognition method based on lip movements, comprising:

[0007] S1. Construct a lip movement dataset based on event data;

[0008] S2. Construct a speaker recognition model based on lip movements;

[0009] The model comprises a four-stage processing procedure, as detailed below:

[0010] Preprocessing stage:

[0011] Perform event-level denoising to suppress spurious events and apply robust enhancements to mitigate the effects of scene changes and noise;

[0012] The original event stream consisting of N events is defined as follows:

[0013] ;

[0014] in, Indicates events on the sensor array spatial coordinates, Represents a timestamp in microseconds. Indicates polarity and is used to indicate increases or decreases in brightness in the sensing environment. Represents the original event stream;

[0015] Event-level denoising: employing a spatiotemporal filter: only when an event has at least one obvious neighboring event within a local 4-connected spatial neighborhood, and the time window is set. Only those that are allowed to be retained will be defined as:

[0016] ;

[0017] Then the noise reduction stream The method is as follows:

[0018] ;

[0019] Enhanced robustness:

[0020] Only perform the following three enhancement strategies during model training:

[0021] Spatial translation strategy:

[0022] Random spatial translations are introduced to simulate slight viewpoint shifts, with the spatial shifts distributed uniformly over the interval [-20, 20]. Mid-sampling, i.e. The event stream after spatial offset processing The offset only retains events that fall within the effective sensor resolution (W, H):

[0023] ;

[0024] ;

[0025] Horizontal mirroring strategy:

[0026] A horizontal mirroring process with a probability of 0.5 was used to simulate symmetrical lip movements. The coordinate system transformation method is as follows:

[0027] ;

[0028] in, These are random variables drawn from a uniform distribution, thus yielding a mirror set. Events that exceed the sensor's boundary range after transformation will be discarded;

[0029] Event sparsity strategy:

[0030] Randomly remove a portion of events from the processed data stream. Sampling discard rate and determine the target count value. , Ultimately enhance the flow By without replacement from Obtained by uniformly sampling T events:

[0031] .

[0032] Time-aware voxel encoding stage:

[0033] The modal gap is bridged by converting sparse asynchronous events into dense and information-rich tensors, and key spatiotemporal cues are preserved through learnable temporal allocation, local spatial aggregation, and temporal channel reweighting.

[0034] Time normalization:

[0035] First, normalize the event timestamp to the range [0, 1]:

[0036] ;

[0037] in, and These represent the first and last timestamps in the sequence, respectively.

[0038] Time available for learning:

[0039] Employing a learnable time allocation mechanism, the first step is to calculate each event. Time offset relative to the starting point of each interval :

[0040] ;

[0041] in, Indicates the number of time intervals;

[0042] Next, a lightweight two-layer, multi-layer sensing approach is used to determine the allocation weights. :

[0043] ;

[0044] Assign weights Indicates an event Contribution to b This represents a learnable scaling parameter used to adjust the sensitivity to time offset;

[0045] The weights are generated by a lightweight two-layer multilayer sensing mechanism, which uses... For input, use a 32-unit ReLU activation function hidden layer and output a scalar weight;

[0046] Weights are assigned based on the calculation of all events. Local spatial aggregation is performed to construct a dense voxel tensor, which aggregates the weighted contribution space of events into a 2B feature map and explicitly separates positive and negative polarities. Specifically, for each polarity... For the temporal interval b, the pixel value at spatial coordinates (x, y) is obtained by summing the assigned weights of all valid events located at that position, resulting in the final bipolar voxel tensor. ;

[0047] Local spatial aggregation:

[0048] Local spatial aggregation is introduced during voxelization, where each voxel is updated by accumulating the values ​​of its directly 2-connected neighbors, thereby enhancing structural connectivity.

[0049] ;

[0050] in, This represents the channel index; indices that exceed the boundary will be discarded. Local spatial continuity is effectively captured by aggregating only a subset of the target neighborhood.

[0051] Time channel reweighting:

[0052] A time compression mechanism is introduced to highlight the information-rich time intervals in an adaptive manner, while suppressing redundant or noisy time intervals.

[0053] First, for each The feature maps undergo global average pooling, which aggregates the global spatial context and converts the voxel tensors... Spatial dimensions compressed into channel descriptors ;

[0054] Subsequently, a one-dimensional convolutional layer is used to capture the local interaction relationships between adjacent time partitions, and attention calibration weights are generated using the Sigmoid activation function.

[0055] ;

[0056] in, This represents the Sigmoid activation function. Indicates calibration weights;

[0057] Through channel-level multiplication Characteristics of primitive voxels Perform recalibration:

[0058] ;

[0059] in, Indicates the calibrated voxel characteristics;

[0060] Structure-aware spatial enhancement stage: Voxelized features are refined through channel compression layers and orientation-aware spatial smoothing to reduce redundancy and enhance spatial representation;

[0061] Channel compression:

[0062] Applying pointwise 1×1 convolution to linearly project high-dimensional channel features into a low-dimensional space:

[0063] ;

[0064] Direction perception spatial smoothness:

[0065] A one-dimensional convolution along the depth direction is applied to the horizontal axis of the feature map, smoothing the horizontal ripples by operating independently on each channel. The smoothed features are then normalized and activated.

[0066] ;

[0067] in, This represents a depth-accessible one-dimensional convolution with a kernel size of 3, zero-padding size of 1, and is applied only along the horizontal direction.

[0068] After spatial smoothing, a channel attention mechanism is employed to adaptively recalibrate based on the enhanced feature channels. Then, a one-dimensional convolution is used to generate channel-level attention weights:

[0069] ;

[0070] ;

[0071] in, This indicates the learning importance of each smoothing channel. This represents the final refined tensor. , Indicates global average pooling;

[0072] Classification and polarity consistency regularization processing stage:

[0073] The refined features are input into the improved ResNet34 backbone network, and polarity consistency regularization is combined to explicitly preserve the motion direction information carried by the event polarity.

[0074] Polarity consistency regularization:

[0075] Using a lightweight convolutional reconstruction head, Mapping back to dual-channel polarity diagram Corresponding to negative and positive polarities respectively, a two-layer CNN structure is used. First, the features are projected into a higher-dimensional space to capture richer interactions, and then mapped to the target dual-channel polarity output:

[0076] ;

[0077] in, Represents two-dimensional convolution.

[0078] Through average initial voxel encoding Mid-polarity specific time channels, deriving the reference tensor ,make Mark them separately The negative and positive polarity partitions, and the formula for calculating the reference polarity diagram are:

[0079] ;

[0080] Spatial normalization was applied to both the reconstructed polar diagram and the reference polar diagram.

[0081] ;

[0082] Then, PCR loss is defined as the difference between the normalized distribution of the reconstructed polarity map and the reference polarity map. distance:

[0083] ;

[0084] By minimizing This encourages the network to retain polarity-aware motion information in the features it learns;

[0085] ResNet34 backbone and total loss:

[0086] The initial 7×7 convolutional layer with stride 2 was replaced with a 3×3 convolutional layer with stride 1, and configured to receive C-channel input from the structure-aware spatial enhancement module.

[0087] Replace the step-size 2 max pooling layer with an identity mapping layer;

[0088] The overall objective function is defined as follows:

[0089] ;

[0090] in, For cross loss, It is a balancing hyperparameter.

[0091] S3. Train the lip movement-based speaker recognition model using an event-based lip movement dataset;

[0092] S4. Finally, the real-time captured dynamic images of the lips are input into the trained speaker recognition model based on lip movements for speaker recognition.

[0093] Furthermore, step S1 specifically includes:

[0094] Event data collection:

[0095] A dual-camera system was used to capture dynamic images of the lips. The two image sensors of the dual-camera system were rigidly connected together to ensure that data from the same scene was collected synchronously. Participants sat at a set distance from the cameras and were asked to maintain a stable posture. Visual stimuli were displayed on a monitor behind the cameras, randomly presenting numbers 0-9 at set intervals. Participants were asked to pronounce each number naturally and clearly. Data was then collected under various lighting intensities and different viewing angles, and the event data collection results were retained.

[0096] Event data annotation:

[0097] The regions of interest containing the lips are labeled, and each sample space is cropped to a resolution of a set number of pixels, retaining only events that fall within this window. The start timestamp of each phonation is also labeled. Based on the timestamps and labels, event segments corresponding to each digit phonation are extracted from the continuous raw event stream. Finally, each segment is assigned a corresponding digit label, thus obtaining a time-aligned event-based lip motion dataset.

[0098] The beneficial effects of this invention are as follows:

[0099] Compared with existing speaker recognition methods based on frame images or static lip appearance features, this invention acquires lip motion event streams using an event camera and, through event-level denoising, time-aware voxel encoding, structure-aware spatial enhancement, and polarity consistency regularization, can more effectively extract dynamic spatiotemporal features of the lips that reflect identity differences, thus having the following beneficial effects:

[0100] Event-level denoising can filter out isolated false events in low-light or complex environments, improve the signal-to-noise ratio of the input event stream, and thus enhance the stability of the recognition process.

[0101] By learningable temporal allocation, local spatial aggregation, and temporal channel reweighting, richer fine-grained temporal information and local spatial continuity can be preserved during voxelization, reducing the loss of temporal information caused by traditional hard-bucket voxelization.

[0102] By using channel compression and orientation-aware spatial smoothing, redundant information and high-frequency noise can be suppressed, spatial representation related to lip movement structures can be enhanced, and feature expression capabilities can be improved.

[0103] By introducing polarity consistency regularization, the motion direction information represented by the event polarity can be preserved, thereby improving the model's ability to depict dynamic changes in the lips.

[0104] By employing spatial translation, horizontal mirroring, and event sparsity enhancement strategies during the training phase, the model's adaptability to changes in viewpoint, lighting, and event sparsity conditions can be improved, thereby enhancing its robustness in cross-scene recognition.

[0105] During the training phase, lip-feature-based identity representations are learned from a single scene, while ensuring robustness to unknown changes in viewpoint and lighting conditions during testing.

[0106] Therefore, the present invention can achieve relatively stable speaker recognition under different lighting conditions and different viewing angles, and has good environmental adaptability and application value. Attached Figure Description

[0107] Figure 1 This is a flowchart of a speaker recognition method based on lip movements provided in an embodiment of the present invention. Detailed Implementation

[0108] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0109] This invention provides a speaker recognition method based on lip movements, such as... Figure 1 As shown, it specifically includes:

[0110] S1. Construct an event-based lip movement dataset;

[0111] To address the issue of insufficient resources for cross-scenario analysis, this invention designs a novel and comprehensive event-based lip movement dataset. This dataset is specifically designed to support identity recognition in challenging real-world scenarios, incorporating systematic variations in viewpoint and lighting conditions. The design principles, acquisition settings, processing flow, and statistical characteristics of this dataset are detailed below.

[0112] Design paradigm:

[0113] Existing lip-related datasets typically follow three paradigms: digit-based, word-based, or sentence-based. The digit paradigm, widely used in identity recognition tasks, requires subjects to pronounce digits with consistent and simple speech structures; it has been adopted by datasets such as XM2VTS, Tulips, qFace, and FAVLIP. The word-based paradigm is suitable for both identity recognition and lip-reading tasks, including MVGL-AVD, LRW, and DVSLip. Sentence-based datasets (such as CREMA-D, TIMIT, GRID, and GRID-CCP) are primarily used for sentiment analysis and lip-reading tasks.

[0114] This invention employs a digit (0-9)-based recognition scheme instead of traditional words or sentences. This design choice is based on three core considerations: minimizing linguistic interference, controlling speech complexity, and expanding practical application scenarios. First, sentence-level and long-word pronunciation can produce co-articulation effects, potentially blurring the subject's unique lip movement patterns, while digit pronunciation has relatively independent and clear pronunciation boundaries. Second, due to the interpretability of deep learning models, complex linguistic content may introduce identity-irrelevant interference factors (such as lip shape features or time interval differences for specific words). Using a digit scheme with uniform speech complexity allows deep learning models to focus on identity-related motion features. Finally, the digit recognition scheme is highly compatible with practical security scenarios such as ID cards, phone numbers, and house numbers.

[0115] Data collection settings:

[0116] Data acquisition is performed using a simultaneous dual-camera system (an event camera and an RGB camera). The observation angle (0°, 45°, 90°) can be adjusted by repositioning the camera along a horizontal arc and maintaining a constant distance from the stationary subject. Illumination intensity is controlled via an indoor lighting system.

[0117] This invention employs a dual-camera system to synchronously acquire visual data. A Prophesee EVK4 event camera (1280×720 resolution) records high temporal resolution lip movements, while a Logitech C922 color camera (1920×1080 resolution, 60 frames / second) acquires RGB reference frames. Both image sensors are rigidly fixed to ensure synchronous data acquisition within the same scene. Subjects are seated 90-110 cm from the cameras and are instructed to maintain a stable posture. Visual stimuli are presented on a monitor behind the cameras, randomly displaying a sequence of digits 0-9 at 3-second intervals. Subjects are required to pronounce each digit naturally and clearly. To alleviate fatigue and maintain pronunciation consistency, a mandatory rest of at least one minute is required after every 50 trials.

[0118] Data acquisition was conducted under four different acquisition conditions to systematically incorporate variations in viewing angle and lighting conditions. Ambient light intensity was quantified using a TASI TA630A photometer, where SI represents “sufficient illumination” (approximately 216 lux) and II represents “insufficient illumination” (approximately 12.5 lux). The dataset of this invention includes four scenarios: SI-0°, SI-45°, SI-90°, and II-0°, corresponding to frontal viewing angles (0°), 45°, and 90° under sufficient illumination (approximately 216 lux), and a low-light frontal viewing angle (0°) under insufficient illumination (approximately 12.5 lux), respectively. Under each condition, this invention collected 100 valid samples from each subject to ensure the comprehensiveness and balance of the dataset, facilitating cross-scenario evaluation.

[0119] Event data organization and annotation:

[0120] To facilitate manual annotation and quality verification, this invention first uses the Metavision SDK (version 5.0) to convert the raw event data captured by the EVK4 sensor into visual frames. Based on these frame images, regions of interest (ROIs) containing the lips are manually annotated, and each sample is spatially cropped to a resolution of 200×160 pixels, retaining only the event data within that window. To achieve accurate temporal segmentation, this invention also annotates the start timestamp of each lip movement. Based on these timestamps and annotation information, 3-second event segments corresponding to individual lip movements are extracted from the continuous raw event stream. Finally, a specific finger label (0-9) is assigned to each segment, thereby constructing a time-aligned and structurally clear event-based lip movement dataset.

[0121] S2. Construct a speaker recognition model based on lip movements;

[0122] The model comprises a four-stage processing procedure, as detailed below:

[0123] Preprocessing stage:

[0124] The raw event stream itself contains background noise, especially under low-light conditions, where spurious events proliferate due to sensor noise. The preprocessing stage aims to improve the signal-to-noise ratio and enhance the model's generalization ability through two key operations: event-level denoising and robustness enhancement. Importantly, denoising is applied directly to the raw event stream, rather than reconstructing frames, to prevent noise amplification during encryption. Furthermore, enhancement is strictly limited to the training phase to ensure the fairness of the evaluation.

[0125] Perform event-level denoising to suppress spurious events and apply robust enhancements to mitigate the effects of scene changes and noise;

[0126] The original event stream consisting of N events is defined as follows:

[0127] ;

[0128] in, Indicates events on the sensor array spatial coordinates, Represents a timestamp in microseconds. Indicates polarity and is used to indicate increases or decreases in brightness in the sensing environment. Represents the original event stream;

[0129] Event-level denoising: Thermal noise or ambient light fluctuations often trigger isolated events, thus disrupting the spatiotemporal coherence of motion patterns. Based on this, this invention employs a spatiotemporal filter: an event is only denoted if it has at least one clearly neighboring event within a locally 4-connected spatial neighborhood, and the time window is set. It effectively balances noise suppression and preservation of effective motion signals.

[0130] The neighborhood of a 4-connected space is defined as:

[0131] ;

[0132] Then the noise reduction stream The method is as follows:

[0133] ;

[0134] This invention No polarity constraints are imposed: Regardless of polarity, adjacent cells can serve as valid evidence of local spatiotemporal activity.

[0135] Robustness Enhancement: To mitigate overfitting and improve robustness to changes in viewpoint and lighting, this invention applies three enhancement strategies only during the training phase. Unless otherwise stated, these operations are performed in sequence: spatial translation, horizontal mirroring, and event sparsification.

[0136] Spatial translation strategy:

[0137] This invention uses spatial translation to simulate slight angular displacement, with the spatial displacement being uniformly distributed over the interval [-20, 20]. Mid-sampling, i.e. The event stream after spatial offset processing The offset only retains events that fall within the effective sensor resolution (W, H):

[0138] ;

[0139] .

[0140] Horizontal mirroring strategy:

[0141] This invention uses a horizontal mirroring process with a probability of 0.5 to simulate symmetrical lip movements. The coordinate system transformation method is as follows:

[0142] ;

[0143] in, These are random variables drawn from a uniform distribution, thus yielding a mirror set. Events that exceed the sensor's boundary range after transformation will be discarded.

[0144] Event sparsity strategy:

[0145] To simulate the sparsity characteristics under low-light conditions, this invention randomly removes a portion of events from the processed data stream. Sampling discard rate and determine the target count value. , Ultimately enhance the flow By without replacement from Obtained by uniformly sampling T events:

[0146] ;

[0147] To prevent excessive information loss and thus avoid erasing crucial motion cues, this sparsification step is applied only to 30% of the training samples. This invention adjusts the sparsification probability in the validation set partitioning, finding that applying it to 30% of the training samples achieves the optimal balance: a higher probability leads to excessive input sparsity and degrades performance, while a lower probability fails to provide sufficient robustness gains.

[0148] Time-aware voxel encoding stage:

[0149] The modal gap is bridged by converting sparse asynchronous events into dense and information-rich tensors, and key spatiotemporal cues are preserved through learnable temporal allocation, local spatial aggregation, and temporal channel reweighting.

[0150] Time normalization:

[0151] To eliminate the dependency on variable record duration, this invention first normalizes the event timestamp to the range [0, 1]:

[0152] ;

[0153] in, and These represent the first (i=1) and last (i=T) timestamps in the sequence, respectively;

[0154] Time available for learning:

[0155] Voxelization is a widely used event representation technique that maps sequential asynchronous events to a discrete, grid-based structure. Standard voxelization typically discretizes the time domain into fixed time intervals and accumulates event polarity counts within each interval to form a dense representation. However, most existing methods quantize sequential timestamps by discarding fine-grained temporal information by hard-assigning each event to a single specific time interval.

[0156] To address this quantification loss, this invention employs a learnable time allocation mechanism. Instead of a rigid allocation, this mechanism allows each event to contribute to multiple time periods through learnable weights, thereby preserving fine-grained temporal dynamics.

[0157] First, calculate each event. Time offset relative to the starting point of each interval :

[0158] ;

[0159] in, Indicates the number of time intervals;

[0160] Next, a lightweight two-layer, multi-layer sensing approach is used to determine the allocation weights. :

[0161] ;

[0162] Assign weights Indicates an event Contribution to b This represents a learnable scaling parameter used to adjust sensitivity to time offsets. Specifically, a larger... The value amplifies smaller offsets, resulting in a sharper time response; while smaller values ​​amplify smaller offsets. The value helps to achieve a smoother weight distribution over a time interval.

[0163] The weights are generated by a lightweight two-layer multilayer sensing mechanism, which uses... The input is a 32-unit ReLU activation function hidden layer, and the output is a scalar weight. This pointwise regressor was chosen to approximate common temporal interpolation kernels flexibly enough while maintaining encoder parameters and computational efficiency.

[0164] It is important to note that lightweight two-layer, multilayer perceptron generation can also be implemented using other pointwise mapping methods, such as CNN-based encoders. Explicitly introducing the interval index b enables interval-dependent temporal assignment, allowing different temporal intervals to emphasize different stages of motion dynamics without introducing any spatial coupling.

[0165] Weights are assigned based on the calculation of all events. Local spatial aggregation is performed to construct a dense voxel tensor, which aggregates the weighted contribution space of events into a 2×2 feature map and explicitly separates positive and negative polarities. Specifically, for each polarity... For the temporal interval b, the pixel value at spatial coordinates (x, y) is obtained by summing the assigned weights of all valid events located at that position, resulting in the final bipolar voxel tensor. By using learned weights instead of binary counts, this aggregation strategy directly embeds precise sub-interval temporal dynamics into feature amplitudes, thereby avoiding the loss of fine-grained motion cues common in standard voxelization.

[0166] Local spatial aggregation:

[0167] The event flow exhibits strong spatial correlations between adjacent pixels, where meaningful structures such as lip contours arise from local continuity rather than isolated pixels. To leverage this dependency, this invention introduces local spatial aggregation during voxelization, updating each voxel by accumulating the values ​​of its directly 2-connected neighbors, thereby enhancing structural connectivity.

[0168] ;

[0169] in, This represents the channel index; indexes that exceed the boundary will be discarded.

[0170] This design only aggregates a subset of the target neighborhood (asymmetric offset), which can effectively capture local spatial continuity and avoid the problems of excessive blurring and redundant accumulation caused by using a large symmetric smoothing kernel.

[0171] Time channel reweighting:

[0172] Different time intervals may contribute differently to identity recognition. To address this issue, this invention introduces time channel reweighting, which adaptively highlights information-rich time intervals while suppressing redundant or noisy time intervals.

[0173] First, for each The feature maps undergo global average pooling, which aggregates the global spatial context and converts the voxel tensors... Spatial dimensions compressed into channel descriptors ;

[0174] Subsequently, a one-dimensional convolutional layer is used to capture the local interaction relationships between adjacent time partitions, and attention calibration weights are generated using the Sigmoid activation function.

[0175] ;

[0176] in, This represents the Sigmoid activation function. Indicates calibration weights;

[0177] Through channel-level multiplication Characteristics of primitive voxels Perform recalibration:

[0178] ;

[0179] in, Indicates the calibrated voxel characteristics;

[0180] By learning the global correlation between polarity-aware time intervals, temporal channel reweighting can effectively highlight key motion phases without changing spatial resolution or introducing a large amount of temporal convolution.

[0181] Structure-aware spatial enhancement stage:

[0182] Voxelized features are refined through channel compression layers and orientation-aware spatial smoothing to reduce redundancy and enhance spatial representation;

[0183] Channel compression layer: voxel tensors generated through time-aware voxel encoding Containing 2B channels, the time intervals and polarities are explicitly separated, which may introduce redundancy between channels. To refine a compact yet expressive spatially enhanced representation, this invention first employs channel-layer compression.

[0184] Specifically, pointwise 1×1 convolution is applied to linearly project high-dimensional channel features into a low-dimensional space:

[0185] ;

[0186] This operation reduces the channel dimension from 2B to C, thereby facilitating the fusion of information from different time intervals and polarities. Empirical evidence shows that this dimensionality reduction achieves an optimal balance between preserving discriminative lip movement cues and suppressing task-irrelevant noise. Unlike attention-based temporal channel reweighting modules, this layer focuses on pixel-wise channel blending and dimensionality reduction, thus improving the computational efficiency of subsequent spatial smoothing stages.

[0187] Direction perception spatial smoothness:

[0188] In lip movement analysis, vertically structured cues (such as the opening and closing boundaries of the lips) are often more informative than purely horizontal variations. To preserve these critical vertical structures while suppressing high-frequency horizontal noise, this invention introduces orientation-aware spatial smoothing.

[0189] Specifically, a one-dimensional convolution along the depth direction is applied along the horizontal axis of the feature map. By operating on each channel independently, this step can smooth out horizontal ripples without obscuring the information-rich vertical structural continuity.

[0190] Smooth the horizontal fluctuations, then normalize and activate the smoothed features:

[0191] ;

[0192] in, The first method represents a depth-accessible one-dimensional convolution with a kernel size of 3 and zero padding of 1, applied only in the horizontal direction. This invention sets the kernel size to 3 and the padding size to 1 to achieve minimal effective local smoothing while maintaining feature map resolution. This helps suppress high-frequency horizontal noise without excessively blurring the lip boundaries of vertical structures, and keeps the operation lightweight.

[0193] After spatial smoothing, a channel attention mechanism is employed to adaptively recalibrate based on the enhanced feature channels. Channel-level attention weights are generated using a GAP followed by a one-dimensional convolution.

[0194] ;

[0195] ;

[0196] in, This indicates the learning importance of each smoothing channel. This represents the final refined tensor. It effectively encodes spatiotemporal features with enhanced structural saliency.

[0197] Classification and polarity consistency regularization processing stage:

[0198] This stage uses an improved ResNet34 backbone network (enhanced by polarity consistency regularization) to perform identity classification. While deep networks excel at extracting semantic features, repetitive spatial aggregation and channel blending in structure-aware spatial augmentation can dilute the underlying polarity cues encoding precise motion orientation. Polarity consistency regularization aims to address this issue by enforcing the enhanced feature representation, thereby maintaining a polarity distribution consistent with the original structure-aware spatial augmentation input.

[0199] The refined features are input into the improved ResNet34 backbone network, and polarity consistency regularization is combined to explicitly preserve the motion direction information carried by the event polarity.

[0200] Polarity consistency regularization:

[0201] To estimate the retained polarity information, this invention employs a lightweight convolutional reconstruction head. Mapping back to dual-channel polarity diagram Corresponding to negative and positive polarities respectively, a two-layer CNN structure is used. First, the features are projected into a higher-dimensional space to capture richer interactions, and then mapped to the target dual-channel polarity output:

[0202] ;

[0203] in, This represents a two-dimensional convolution.

[0204] This invention employs a two-layer CNN as the minimum reconstruction head: single-layer convolutions are often insufficient to recover polarity patterns after channel blending and smoothing, while deeper reconstruction heads introduce unnecessary parameters and may lead to overfitting. The expand-compress design provides sufficient capacity to model local interactions before projecting them onto the desired two-channel polarity map. This expand-compress design enhances the reconstruction head's ability to recover fine-grained polarity details from the compressed feature space.

[0205] This invention utilizes average initial voxel encoding Mid-polarity specific time channels, deriving the reference tensor ,make Mark them separately The negative and positive polarity partitions, and the formula for calculating the reference polarity diagram are:

[0206] ;

[0207] Spatial normalization was applied to both the reconstructed polar diagram and the reference polar diagram.

[0208] ;

[0209] Then, the polarity consistency regularization loss. Defined as the normalized distribution between the reconstructed polar diagram and the reference polar diagram. distance:

[0210] ;

[0211] By minimizing This encourages the network to retain polarity-aware motion information in the features it learns;

[0212] ResNet34 backbone and total loss:

[0213] ResNet34 was originally designed for 3-channel RGB images and employed aggressive early downsampling, which could lose fine-grained motion cues crucial for event-based analysis. To adapt the backbone network to the enhanced event representations of this invention, two key modifications are introduced:

[0214] The initial 7×7 convolutional layer with stride 2 was replaced with a 3×3 convolutional layer with stride 1, and configured to receive C-channel input from the structure-aware spatial enhancement module.

[0215] Replace the step-size 2 max pooling layer with an identity mapping layer;

[0216] The above changes preserve the spatial resolution of sparse but discriminative motion patterns in the early stages of the network.

[0217] The overall objective function is defined as follows:

[0218] ;

[0219] in, For cross loss, It is a balancing hyperparameter.

[0220] S3. Train the lip movement-based speaker recognition model using an event-based lip movement dataset;

[0221] S4. Finally, the real-time captured dynamic images of the lips are input into the trained speaker recognition model based on lip movements for speaker recognition.

[0222] The above description is merely a preferred embodiment of the present invention. It should be understood that the present invention is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the concept described herein through the above teachings or related technologies or knowledge. Modifications and variations made by those skilled in the art that do not depart from the spirit and scope of the present invention should be within the protection scope of the appended claims.

Claims

1. A speaker recognition method based on lip movements, characterized in that, include: S1. Construct an event-based lip movement dataset; S2. Construct a speaker recognition model based on lip movements; The model includes the following processing steps: Preprocessing stage: Perform event-level denoising to suppress spurious events and apply robust enhancements to mitigate the effects of scene changes and noise; The original event stream consisting of N events is defined as follows: ； in, Indicates events on the sensor array spatial coordinates, Represents a timestamp in microseconds. Indicates polarity and is used to indicate increases or decreases in brightness in the sensing environment. Represents the original event stream; Event-level denoising: using a spatiotemporal filter: only when an event has at least one obvious neighboring event in its local 4-connected spatial neighborhood, and the time window is set to... Only those that are allowed to be retained will be defined as: ； Then the noise reduction stream The method is as follows: ； Time-aware voxel encoding stage: The impact of quantization error is mitigated by converting sparse asynchronous events into dense and information-rich tensors, and key spatiotemporal cues are preserved through learnable temporal allocation, local spatial aggregation, and temporal channel reweighting. Time normalization: First, normalize the event timestamp to the range [0, 1]: ； in, and These represent the first and last timestamps in the sequence, respectively. Structure-aware spatial enhancement stage: The structure-aware space enhancement stage resolves the retained channel redundancy and local spatial anomalies caused by residual noise through channel compression and orientation-aware space smoothing. Classification and polarity consistency regularization processing stage: The refined features are input into the improved ResNet34 backbone network, and polarity consistency regularization is combined to explicitly preserve the motion direction information carried by the event polarity. Polarity consistency regularization: Using a lightweight convolutional reconstruction head, Mapping back to dual-channel polarity diagram The two channels correspond to negative and positive polarities respectively. A two-layer CNN structure is used, first projecting the features into a higher-dimensional space to capture richer interactions, and then mapping them to the target dual-channel polarity output. ； in, Represents two-dimensional convolution. This represents the final refined tensor; Through average initial voxel encoding Mid-polarity specific time channels, deriving the reference tensor ,make Mark them separately The negative and positive polarity partitions, and the formula for calculating the reference polarity diagram are: ； In the formula, Indicates the number of time intervals; Spatial normalization was applied to both the reconstructed polarity diagram and the reference polarity diagram. ； Then, PCR loss is defined as the difference between the normalized distribution of the reconstructed polarity map and the reference polarity map. distance: ； By minimizing This encourages the network to retain polarity-aware motion information in the features it learns; ResNet34 backbone and total loss: The initial 7×7 convolutional layer with stride 2 was replaced with a 3×3 convolutional layer with stride 1, and configured to receive C-channel input from the structure-aware spatial enhancement module; Replace the step-size 2 max pooling layer with an identity mapping layer; The overall objective function is defined as follows: ； in, For cross-entropy loss, It is a balancing hyperparameter; S3. Train the lip movement-based speaker recognition model using an event-based lip movement dataset; S4. Finally, the real-time captured dynamic images of the lips are input into the trained speaker recognition model based on lip movements for speaker recognition.

2. The speaker recognition method based on lip movements according to claim 1, characterized in that, After event-level denoising, the preprocessing stage also includes robustness enhancement, the specific process of which is as follows: Only perform the following three enhancement strategies during model training: Spatial translation strategy: Random spatial translations are introduced to simulate slight viewpoint shifts, with the spatial shift amount varying from the interval... uniform distribution on Mid-sampling, i.e. The event stream after spatial offset processing The offset only retains events that fall within the effective sensor resolution (W, H): ； ； Horizontal mirroring strategy: A horizontal mirroring process with a probability of 0.5 was used to simulate symmetrical lip movements. The coordinate system transformation method is as follows: ； in, These are random variables drawn from a uniform distribution, thus yielding a mirror set. Events that exceed the sensor's boundary range after transformation will be discarded; Event sparsity strategy: Randomly remove a portion of events from the processed data stream. Sampling discard rate and determine the target count value. , Ultimately enhance the flow By without replacement from Obtained by uniformly sampling T events: 。 3. The speaker recognition method based on lip movements according to claim 1, characterized in that, Step S1 specifically includes: Event data collection: A dual-camera system was used to capture dynamic images of the lips. The two image sensors of the dual-camera system were rigidly connected together to ensure that data from the same scene was collected synchronously. Participants sat at a set distance from the cameras and were asked to maintain a stable posture. Visual stimuli were displayed on a monitor behind the cameras, randomly presenting numbers 0-9 at set intervals. Participants were asked to pronounce each number naturally and clearly. Data was then collected under various lighting intensities and different viewing angles, and the event data collection results were retained. Event data annotation: The regions of interest containing the lips are labeled, and each sample space is cropped to a resolution of a set number of pixels, retaining only events that fall within this window. The start timestamp of each phonation is also labeled. Based on the timestamps and labels, event segments corresponding to each digit phonation are extracted from the continuous raw event stream. Finally, each segment is assigned a corresponding digit label, thus obtaining a time-aligned event-based lip motion dataset.

4. The speaker recognition method based on lip movements according to claim 1, characterized in that, The time-aware voxel encoding stage includes learnable time allocations, which are detailed below: A learnable time allocation mechanism is adopted. First, the interval is evenly divided into B equal parts, and the calculation is performed for each event. Time offset relative to the starting point of each interval : ； in, Indicates the number of time intervals; Next, a lightweight two-layer, multi-layer sensing approach is used to determine the allocation weights. : ； Assign weights Indicates an event Contribution to b This represents a learnable scaling parameter used to adjust the sensitivity to time offset.

5. The speaker recognition method based on lip movements according to claim 4, characterized in that, The weights are generated by a lightweight two-layer multilayer sensing mechanism, which uses... The input is a 32-unit ReLU activation function hidden layer, and the output is a scalar weight.

6. The speaker recognition method based on lip movements according to claim 4, characterized in that, The learnable time allocation mechanism also includes: calculating and allocating weights based on all events. Local spatial aggregation is performed to construct a dense voxel tensor, which aggregates the weighted contribution space of events into a 2B feature map and explicitly separates positive and negative polarities. Specifically, for each polarity... For the temporal interval b, the pixel value at spatial coordinates (x, y) is obtained by summing the assigned weights of all valid events located at that position, resulting in the final bipolar voxel tensor. , 2 represents two polarities.

7. The speaker recognition method based on lip movements according to claim 6, characterized in that, The time-aware voxel encoding stage also includes local spatial aggregation: Local spatial aggregation is introduced during voxelization, where each voxel is updated by accumulating the values ​​of its directly 2-connected neighbors, thereby enhancing structural connectivity. ； in, This represents the channel index; indices that exceed the boundary will be discarded. Local spatial continuity is effectively captured by aggregating only a subset of the target neighborhood.

8. The speaker recognition method based on lip movements according to claim 6, characterized in that, The time-aware voxel encoding stage also includes time channel reweighting: A time compression mechanism is introduced to highlight the information-rich time intervals in an adaptive manner, while suppressing redundant or noisy time intervals. First, a global average pooling operation is performed on each feature map, and the voxel tensors are aggregated by aggregating the global spatial context. Spatial dimensions compressed into channel descriptors ; Subsequently, a one-dimensional convolutional layer is used to capture the local interaction relationships between adjacent time partitions, and attention calibration weights are generated using the Sigmoid activation function. ； in, This represents the Sigmoid activation function. Indicates calibration weights; Through channel-level multiplication Characteristics of primitive voxels Perform recalibration: ； in, Indicates the calibrated voxel characteristics; Structure-aware spatial enhancement stage: Voxelized features are refined through channel compression layers and orientation-aware spatial smoothing to reduce redundancy and enhance spatial representation.

9. The speaker recognition method based on lip movements according to claim 8, characterized in that, The specific process of channel compression is as follows: Applying pointwise 1×1 convolution to linearly project high-dimensional channel features into a low-dimensional space: 。 10. The speaker recognition method based on lip movements according to claim 9, characterized in that, The specific process of spatial smoothing for orientation perception is as follows: A one-dimensional convolution along the depth direction is applied to the horizontal axis of the feature map, smoothing the horizontal ripples by operating independently on each channel. The smoothed features are then normalized and activated. ； in, This represents a depth-accessible one-dimensional convolution with a kernel size of 3, zero-padding size of 1, and is applied only along the horizontal direction. After spatial smoothing, a channel attention mechanism is employed to adaptively recalibrate based on the enhanced feature channels. Channel-level attention weights are generated using a GAP followed by a one-dimensional convolution. ； ； in, This indicates the learning importance of each smoothing channel. This represents the final refined tensor. .

Images