Federal lip language recognition method and system
By training a lip-reading model on a local device using federated learning and employing encryption technology to protect privacy, the problems of data scarcity and privacy leakage are solved, achieving efficient lip-reading recognition suitable for resource-constrained environments.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-15
- Publication Date
- 2026-03-13
AI Technical Summary
Existing lip-reading systems suffer from data scarcity, privacy risks, and high communication overhead. In particular, high-quality lip movement data is expensive to acquire and difficult to annotate, and personal facial information is easily leaked. Data transmission also leads to high bandwidth and storage costs.
By employing federated learning, lip movement data is stored on local devices, and encrypted model parameters are distributed and aggregated through a central server for training. Differential privacy and homomorphic encryption technologies are used to protect privacy, and personalized model adjustment mechanisms and model distillation techniques are combined to improve recognition accuracy.
It solves the problems of data scarcity and privacy leakage, reduces communication and storage costs, improves the model's generalization ability and recognition accuracy, and is suitable for resource-constrained scenarios.
Smart Images
Figure CN121662045A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer vision technology, specifically to a federated lip-reading method and system. Background Technology
[0002] Lip reading, also known as visual speech recognition, is a language understanding technology that analyzes visual information such as the speaker's lip movements and morphological changes to identify the content the speaker is expressing without relying on audio signals. Essentially, it converts visual lip movements (such as the lips, chin, and tongue) into corresponding speech content (such as phonemes, words, or sentences). Lip reading has broad application potential in noisy environments, far-field speech recognition, and assisting the hearing impaired. A typical deep learning-based lip reading process includes: inputting a video frame sequence of the speaker's face, detecting and extracting the lip region, extracting visual features, time series modeling, and classification and decoding.
[0003] Existing lip-reading systems primarily rely on centralized data training, typically depending on large-scale, manually annotated lip movement datasets for model training. These systems usually require collecting substantial amounts of personal facial information and lip movement video data, uploading this data centrally to a server, and then utilizing the server's powerful computing capabilities for feature extraction, model training, and other operations to ultimately achieve lip-reading recognition. This centralized training method suffers from the following problems: Data scarcity: High-quality lip movement data is costly to acquire and difficult to annotate, resulting in limited available training data. The number of open lip movement datasets is limited, and their coverage of languages, pronunciation scenarios, and facial expression variations is insufficient, leading to inadequate generalization ability of the models. Privacy risks: Centralized training requires users to upload their personal facial images to the cloud, posing a serious risk of privacy breaches, especially when the data contains sensitive information.
[0004] High communication overhead: Transmitting large amounts of facial video data to the cloud results in high bandwidth and storage costs. Summary of the Invention
[0005] To address some or all of the technical problems existing in the prior art, the present invention provides a federal lip-reading method and system.
[0006] The technical solution of the present invention is as follows: Firstly, a federated lip-reading method is provided, which includes: The central server initializes a global lip-reading model and distributes the global model to multiple local clients; Each local client trains the global model locally based on local lip movement data, generates local model parameters, and uploads the parameters to the central server after encrypting them using differential privacy mechanism and homomorphic encryption technology. The central server aggregates the encrypted parameters from all clients and generates updated global model parameters, then feeds back the updated global parameters to each local client. The local client adjusts the local model based on the updated global parameters and fine-tunes the local model through a personalized model adjustment mechanism; Repeat the above steps until the model converges, and finally implement lip reading recognition on the local client.
[0007] In one embodiment of the present invention, the global lip reading recognition model is a sequence modeling network based on deep learning, which adopts an end-to-end learning framework based on connection-time classification loss function to maintain global system labels in order to support lip reading recognition of different languages.
[0008] In one embodiment of the present invention, encrypting the parameters using differential privacy mechanism and homomorphic encryption technology further includes: During the local model training phase, noise is added to the gradient or model parameters to implement differential privacy. The uploaded model parameters are encrypted using homomorphic encryption technology to prevent the central server from decrypting and obtaining the original data.
[0009] In one embodiment of the present invention, the central server uses a federated averaging algorithm and model distillation technique to generate updated global model parameters, wherein the federated averaging algorithm weights the aggregation based on the amount of local data on the client.
[0010] In one embodiment of the present invention, the federated averaging algorithm is further combined with a dynamic aggregation mechanism to adjust the weights based on client data quality, model update magnitude, or historical contribution.
[0011] In one embodiment of the present invention, the central server generates updated global model parameters using a federated averaging algorithm and model distillation techniques, further comprising: The central server treats the local models of multiple clients as "teacher models" and generates soft targets through model distillation techniques; The "student model" is trained based on the soft target as the updated global model, and the global parameters are optimized by minimizing the KL divergence loss.
[0012] In one embodiment of the present invention, the personalized model adjustment mechanism includes: The local client fine-tunes the global model based on its own lip movement data to generate a local model that adapts to the user's lip movement characteristics; The fine-tuning process preserves the global knowledge generated by federated learning while adapting to individual differences in lip movement characteristics among users.
[0013] In one embodiment of the present invention, prior to the local training, the client performs data preprocessing, including: The local lip movement video is normalized, noise is filtered, and the lip region is extracted and cropped to generate standardized input.
[0014] In one embodiment of the present invention, the method further includes a dynamic reward mechanism, which allocates rewards based on the client's contribution, the contribution being evaluated through model update quality or data volume.
[0015] Secondly, embodiments of the present invention provide a federated lip-reading system, comprising: The central server is used to initialize the global lip reading model, distribute the model to local clients, and aggregate the encrypted local model parameters to generate updated global model parameters. Multiple local clients are used to train the global model locally based on local lip movement data, generate local model parameters, and upload them to the central server through an encryption module. An encryption module, integrated into the local client, is used to implement differential privacy and homomorphic encryption on local model parameters; The aggregation module, integrated into the central server, is used to aggregate encrypted local model parameters to generate updated global model parameters. The personalization adjustment module, integrated into the local client, is used to fine-tune the global model to adapt it to individual user characteristics.
[0016] The main advantages of the technical solution of this invention are as follows: The federated lip-reading recognition method and system of this invention utilizes federated learning to distribute and store lip movement data from multiple users across their respective devices, avoiding centralized data collection and transmission and solving the problem of data scarcity. Federated learning updates the model only on the local device; users' lip movement data remains locally and is never uploaded to the cloud, ensuring personal privacy. Federated learning only transmits model parameters (which are usually encrypted), rather than the raw data, significantly reducing communication and storage costs, making it suitable for resource-constrained scenarios. By jointly updating the local models of multiple users, federated learning can aggregate global knowledge, improving the model's generalization ability and recognition accuracy, especially when there are significant differences between the data from different users. Attached Figure Description
[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 This is a flowchart of a federalized lip-reading recognition method according to an embodiment of the present invention; Figure 2 This is a structural diagram of the LipNet model in a federated lip-reading method according to an embodiment of the present invention; Figure 3 This is a structural diagram of a federal lip-reading system according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the federated learning process of a federated lip-reading recognition method according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the global aggregation process based on model distillation in a federalized lip reading method according to an embodiment of the present invention. Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0020] The technical solutions provided by the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.
[0021] This invention provides a federalized lip-reading method, as shown in the attached figure. Figure 1 As shown, it includes: S101, the central server initializes the global lip reading recognition model and distributes the global model to multiple local clients.
[0022] The federated lip-reading method employs a federated learning framework, as shown in the attached diagram. Figure 3 As shown, it includes a central server and multiple local clients.
[0023] In this step, the central server constructs a deep learning-based lip-reading recognition model. This model employs a sequence modeling framework, capable of processing the temporal features of lip movement videos and outputting the corresponding text content. The model's initial parameters can be pre-trained using publicly available, non-sensitive lip movement datasets (such as the LRW dataset) to ensure basic recognition capabilities.
[0024] The central server can distribute the initialized global model to multiple local clients (such as smartphones, smart home devices, etc.) via secure communication protocols (such as HTTPS). Each client independently stores the model and prepares it for local training.
[0025] S102, each local client trains the global model locally based on local lip movement data, generates local model parameters, and uploads the parameters to the central server after encrypting them through differential privacy mechanism and homomorphic encryption technology.
[0026] After downloading the global model, the client performs local training using local lip movement data (such as 100 lip movement videos recorded by the user). For example, the training configuration is as follows: a stochastic gradient descent (SGD) optimizer is used, the learning rate is set to 0.001, and each client trains for 50 epochs.
[0027] In this embodiment of the invention, user privacy is protected through differential privacy and homomorphic encryption. For example, the specific implementation of differential privacy can be as follows: during the model training phase, Gaussian noise is added to the gradient update to ensure that the information of a single sample cannot be reversed. For example, the specific implementation of homomorphic encryption can be as follows: local model parameters (such as fully connected layer weights) are encrypted using the Paillier homomorphic encryption algorithm, and the generated ciphertext parameters are then uploaded to the central server.
[0028] S103, the central server aggregates the encrypted parameters of all clients and generates updated global model parameters, and then feeds back the updated global parameters to each local client.
[0029] After receiving the encrypted parameters from all clients, the central server can employ different strategies to weight and aggregate the ciphertext parameters. This embodiment does not impose specific limitations on this approach. For example, the weights are allocated based on the proportion of local data volume for each client (e.g., if client A's data volume accounts for 20% of the total, then its parameter weight is 0.2).
[0030] The aggregated global parameters are still returned to each client in encrypted form. After decryption using their private key, the clients merge the decrypted parameters with their local model parameters.
[0031] S104, the local client adjusts the local model based on the updated global parameters and fine-tunes the local model through a personalized model adjustment mechanism.
[0032] The client linearly combines the decrypted global parameters and local parameters according to a preset ratio (e.g., global parameter weight 0.7, local parameter weight 0.3) to generate an updated local model.
[0033] The client can also fine-tune the updated model for an additional two epochs based on local data, focusing on optimizing model layers related to user lip movement features (such as fully connected layers), while keeping the parameters of other layers relatively fixed to avoid the loss of global knowledge.
[0034] S105, repeat steps S102-S104 above until the model converges, and finally implement lip reading recognition on the local client.
[0035] The above steps (local training → encrypted upload → global aggregation → personalized fine-tuning) are executed cyclically until the model converges. For example, the convergence criterion can be: the model is considered converged when the accuracy fluctuation on the validation set is less than 1% over three consecutive training rounds.
[0036] After training, each client uses the final local model to recognize the lip movement videos captured in real time. For example, a user inputs lip movement video through their phone's camera, and the model directly outputs the recognition result on the device (such as "turn on the lights"), without needing to upload the raw data to the cloud.
[0037] In summary, the federated lip-reading recognition method provided in this invention, through federated learning, distributes and stores the lip movement data of multiple users across their respective devices, avoiding centralized data collection and transmission and solving the problem of data scarcity. Federated learning updates the model only on the local device; users' lip movement data remains locally and is never uploaded to the cloud, ensuring the security of personal privacy. Federated learning only needs to transmit model parameters (which are usually encrypted), rather than the raw data, significantly reducing communication and storage costs, making it suitable for resource-constrained scenarios. By jointly updating the local models of multiple users, federated learning can aggregate global knowledge, improving the model's generalization ability and recognition accuracy, especially when there are significant differences between the data of different users.
[0038] In an optional embodiment of the present invention, the global lip reading recognition model is a sequence modeling network based on deep learning, and an end-to-end learning framework based on connection temporal classification loss function is used to maintain global system labels in order to support lip reading recognition of different languages.
[0039] Deep learning-based sequence modeling networks can be a combination architecture of Convolutional Neural Networks (CNN) and Bidirectional Long Short-Term Memory Networks (BiLSTM) for extracting temporal features and performing end-to-end learning. Specifically, this embodiment uses LipNet as the basic architecture, as shown in the attached figure. Figure 2 As shown, it includes the following modules: Spatiotemporal convolutional layer: Extract local spatiotemporal features of lip movement video through 3D convolutional layer (e.g., input size of 30 frames × 64 × 64 × 1).
[0040] Bidirectional recurrent neural network (BiLSTM): performs temporal modeling on the feature sequence output by the convolutional layer to capture the contextual dependencies of lip movements (e.g., using two layers of BiLSTM, each with 128 hidden units).
[0041] CTC decoder: maps the temporal features output by BiLSTM to text sequences, and supports end-to-end training (e.g., the output layer has 36 English labels or 3500 Chinese characters).
[0042] The loss function is trained using the Connectionist Temporal Classification (CTC) loss function, which is adaptable to unaligned sequence data, i.e., it does not require frame-by-frame annotation of lip-reading text. The formula is: ; in, This indicates that all output sequences can be obtained by removing duplicate elements and whitespace labels. The set of paths The actual label sequence, Given the input sequence, Indicates the input sequence The path to be generated The probability of.
[0043] The server maintains a global label space that supports multilingual lip-reading recognition, ensuring that training data from different clients can be mapped within a unified label framework. For example, English uses a label set of 26 letters + 10 numbers; Chinese uses a label set of 3500 commonly used Chinese characters, combined with pinyin or phoneme-level recognition modes. Clients dynamically load the corresponding label mapping table based on their local language requirements; for example, Chinese clients use Chinese character labels, and English clients use letter labels.
[0044] With this setup, this embodiment utilizes an end-to-end learning approach, eliminating the need for manual alignment of video frames and text labels, simplifying the training process and improving the model's generalization ability. Through a unified model architecture and dynamic label mapping, it supports cross-language lip-reading recognition, reducing deployment costs in multilingual scenarios.
[0045] In an optional embodiment of the present invention, the parameters are encrypted using a differential privacy mechanism and homomorphic encryption technology, further including: During the local model training phase, noise is added to the gradients or model parameters to implement differential privacy. Specifically, during local model training, noise is added to the gradient tensor after each gradient update. The noise type can be Gaussian noise, and the noise range can be [missing information]. Noise standard deviation According to privacy budget Dynamic adjustment (e.g.) hour, Through differential privacy budgeting Quantify the risk of privacy breaches to ensure that the impact of a single sample on the model is strictly limited.
[0046] Uploaded model parameters are encrypted using homomorphic encryption to prevent the central server from decrypting and obtaining the original data. For example, the Paillier homomorphic encryption algorithm can be used, supporting addition operations in ciphertext. The parameter encryption steps include: the client generating a public-private key pair, uploading the public key to the server, and storing the private key locally. Local model parameters (such as the weight matrix of fully connected layers) are then encrypted. ,satisfy When performing aggregation on the server side, the server directly calculates a weighted average of the encrypted parameters without decryption.
[0047] In this embodiment, the original lip movement data is always stored locally, and the server only processes encrypted parameters, completely eliminating the risk of privacy leakage. Homomorphic encryption ensures that the parameter content is irreversible during the aggregation process, so even if the server is attacked, it cannot obtain sensitive user information. Differential privacy mechanism and homomorphic encryption technology are seamlessly integrated with mainstream deep learning frameworks (such as PyTorch and TensorFlow), requiring only additional encapsulation of encryption / decryption interfaces, resulting in strong compatibility.
[0048] In an optional embodiment of the present invention, the central server uses a federated averaging algorithm and model distillation technology to generate updated global model parameters, with the federated averaging algorithm weighted by the amount of local data on the client.
[0049] The FedAvg algorithm calculates aggregation weights based on the proportion of local data volume for each client. For example, client A has a data volume of... The total amount of data is Then its weight is .
[0050] The parameter aggregation formula for the central server is: ,in, For the model parameters of the k-th client, The weight corresponding to the period.
[0051] In model distillation, the server uses local models from multiple clients as teacher models to predict a batch of public data (such as unlabeled lip movement videos), generating an average probability distribution (soft target). The global model (student model) is optimized by minimizing the KL divergence loss with the soft target, as shown in the formula: ; in, It is the predicted distribution of the teacher model. It is the predicted distribution of the student model.
[0052] In this embodiment, the federated averaging algorithm alleviates the problem of uneven data distribution through weighted aggregation, improving model robustness. Model distillation integrates knowledge from multiple clients to generate a global model with stronger generalization ability, which helps improve model accuracy. The distillation process is performed only on the server side, requiring no additional computational burden on the client, thus improving computational efficiency.
[0053] In an optional embodiment of the present invention, the federated averaging algorithm is further combined with a dynamic aggregation mechanism to adjust the weights based on client data quality, model update magnitude, or historical contribution.
[0054] When conducting data quality assessment, data quality can be quantified based on the signal-to-noise ratio (SNR) of client data or the label consistency score (such as cross-validation accuracy), with higher quality data receiving greater weight.
[0055] When evaluating the magnitude of model updates, the cosine similarity between the client's local parameters and the global parameters can be calculated. The lower the similarity (the greater the update magnitude), the higher the weight.
[0056] When evaluating historical contributions, the number of training rounds participated in by the client and the contribution value can be recorded, and the weights can be dynamically adjusted using the sliding window averaging method.
[0057] ; in, , , The adjustment coefficients sum to 1. , , These are quantified data quality, model update magnitude, and historical contribution.
[0058] This setup prevents clients with large datasets from monopolizing model updates, encourages high-quality data contributions, and improves fairness. Dynamic weights allow the model to adapt to changes in data distribution more quickly, resulting in faster convergence. Furthermore, assigning low weights to parameters from low-quality or malicious clients enhances the model's robustness against interference and improves its security.
[0059] In an optional embodiment of the present invention, when the central server aggregates client model parameters, it can selectively aggregate only the parameters of the intermediate layers of the LipNet model, or distill only the feature representations of the intermediate layers. The parameters of the speech content decoding module adopt the parameters trained on the central server. This setting reduces data transmission volume and communication costs. Furthermore, since the intermediate layer features are not directly related to the final output (such as text content), it is difficult to infer sensitive user information in reverse, thus providing stronger privacy protection. The intermediate layers learn common features across users, allowing the model to adapt to different lip movement habits, improving generalization ability and recognition accuracy. In addition, the server uniformly optimizes the decoding parameters to avoid model confusion caused by differences in client labels and maintain consistency across multiple languages.
[0060] In an optional embodiment of the present invention, the personalized model adjustment mechanism includes: The local client fine-tunes the global model based on its own lip movement data to generate a local model that adapts to the user's lip movement characteristics; The fine-tuning process preserves the global knowledge generated by federated learning while adapting to individual differences in lip movement characteristics among users.
[0061] Fine-tuning strategies can employ layered fine-tuning and / or learning rate adjustment. Layered fine-tuning means fixing the spatiotemporal convolutional layer parameters (preserving global feature extraction capabilities) and only fine-tuning the BiLSTM and CTC decoder layers. Learning rate adjustment means using a lower learning rate (e.g., 0.0001) to avoid destroying global knowledge.
[0062] Personalized adjustments are made to accommodate individual differences, adjusting the time step weights of the BiLSTM based on user lip movement habits (such as speech rate and mouth size). For example, if user A speaks quickly, the model is made more sensitive to short-term changes in lip movement.
[0063] This setup enables personalized recognition, adapting to diverse user needs. Through a global-local balancing strategy, the adjusted model retains global knowledge while adapting to individual characteristics, avoiding "catastrophic forgetting."
[0064] In an optional embodiment of the present invention, transfer learning techniques can be introduced on top of federated learning, allowing users to transfer knowledge from a pre-trained general model and quickly adapt to new lip movement data. Transfer learning can significantly reduce training time and resource consumption, and is particularly suitable for new users or situations with limited data.
[0065] In federated learning, the client is responsible for local model training and uses local lip-sync data for model updates (such as gradient calculation or parameter optimization). When introducing transfer learning, the client obtains a pre-trained general model from the server and fine-tunes the model based on a small amount of local data (such as adjusting parameters of some network layers) to quickly adapt to new users or small data scenarios. The server is responsible for aggregating client model updates and generating a global model. In transfer learning, the server provides a pre-trained general model (trained on a large-scale public dataset) for clients to download and use.
[0066] In an optional embodiment of the present invention, prior to local training, the client performs data preprocessing, including: The local lip movement video undergoes image normalization, noise filtering, lip region extraction, and cropping to generate standardized input. Standardized input reduces data distribution differences between clients and improves the stability of federated training. A specific data preprocessing workflow is as follows: Image normalization: Scales video frames to 224×224 resolution and normalizes pixel values to the range [0,1].
[0067] Lip region extraction: The lip region (e.g., 68 key points) was located using the MediaPipe facial key point detection model and cropped to a 64×64 ROI.
[0068] Noise Removal: Use temporal median filtering (window size 5 frames) to eliminate sudden changes in lighting or motion blur.
[0069] In an optional embodiment of the present invention, the federated lip-reading method further includes a dynamic reward mechanism that allocates rewards based on client contributions, which are evaluated through model update quality or data volume. To encourage active client participation in training, the incentive mechanism rewards clients based on their contributions. Contribution can be evaluated through factors such as the quality of model updates, the quantity and quality of data, etc. Rewards can take the form of virtual currency, points, etc., to incentivize clients to provide high-quality data or actively participate in training. The incentive mechanism needs to fairly evaluate the contribution of each client to ensure the fairness of reward allocation.
[0070] In an optional embodiment of the present invention, the federated lip-reading recognition method provided by the present invention can adopt a distributed inference architecture based on edge computing. That is, during the inference stage, some computing tasks can be migrated to edge devices, reducing the pressure on the cloud. Edge computing can perform preliminary lip movement feature extraction and preprocessing on local devices, and only transmit key information to the cloud for final recognition, further reducing communication costs and improving real-time performance.
[0071] The federal lip reading method of the present invention will be described in detail below with reference to a specific embodiment.
[0072] This embodiment of the federated lip-reading method employs a federated learning framework, comprising multiple local clients (user devices) and a coordination server (central server). The local clients are distributed across different data owners, such as various terminal devices using lip-reading functionality or different institutional data sources, each responsible for training or fine-tuning the lip-reading model on its local device. The coordination server is responsible for aggregating model updates from multiple clients and coordinating the interactions between clients as well as controlling the overall model training process. The overall framework is as follows: Figure 3 As shown. The federated learning process includes the following steps: S201: Central Server Model Initialization: The central server first constructs a baseline LipNet lip-reading model and maintains a global label space. Each client's label is mapped to this global space. For example, the label set for English corpora is defined as 36 labels (26 letters + 10 numbers), or the label set for Chinese corpora is defined as 3500 commonly used Chinese characters. Regardless of whether the client uses word-level or sentence-level labels, they are uniformly normalized to a classification learning objective based on the CTC (Connectionist Temporal Classification) loss function. The central server can first train the model based on a publicly available, non-sensitive, and privacy-preserving lip movement dataset (or readily available global lip movement data if available), thus completing parameter initialization. Finally, this model is distributed to the participating devices.
[0073] S202: Local Data Preprocessing: On each local client, the acquired video data containing personal facial and lip movements is preprocessed, such as image normalization, extraction and cropping of lip movement regions, etc., to process the raw data into a format suitable for subsequent calculations. During this process, all data is stored and processed only on the local client and is not transmitted externally.
[0074] S203: Local Model Initialization and Training: Each local client downloads the lip-reading recognition model distributed by the central server and performs preliminary model parameter optimization using local data to obtain local model parameters. This stage is performed independently by each client, and its data and model training process occurs entirely within the local environment, without involving the external transmission of private data. To further protect user privacy, a differential privacy mechanism is introduced during model updates. By adding noise to the gradient or model parameters, differential privacy ensures that even if an attacker obtains update information from some clients, they cannot infer the specific lip movement data.
[0075] S204: Encrypted Parameter Upload and Aggregation: After local training, each client uses homomorphic encryption and other encryption techniques to encrypt the parameters of its local model before uploading them to the coordination server. Upon receiving the encrypted parameters from each client, the coordination server uses a federated averaging algorithm (FedAvg) to aggregate these encrypted parameters, generating a comprehensive global model parameter update value. During this process, the coordination server cannot decrypt and see the actual parameters of each client, ensuring privacy. Considering the inconsistent data distribution among different clients, this invention uses multiple client models as "teacher" models, utilizing their prediction results to generate soft targets. The generated soft targets are then used to train a new "student" model, i.e., the new global model. The student model learns from the knowledge of multiple teacher models, integrating the data distribution and features of each client to improve generalization ability. At the central server, multiple client models are used to predict a batch of data, generating soft targets (e.g., averaging the predicted softmax probabilities). The generated soft targets are used to train the global model, updating the global model's parameters by minimizing the difference between the global model's output and the soft targets (e.g., KL divergence loss). The global model aggregation process based on model distillation can be found in [reference needed]. Figure 5 The following describes the specific processes of the federated averaging algorithm and the model-based distillation polymerization process: Let the total number of clients be , No. The local model parameters for each client are . No. The number of data samples owned by each client is Total global data volume Thus, the central server obtains the parameters of the client's local model through weighted average aggregation. : This approach ensures that clients providing larger amounts of data (assuming adequate data quality) contribute more significantly to the global model, reflecting the core idea of federated learning: "data volume dominates aggregation weights."
[0076] Let the original model parameters of the central server be... During distillation, the input sample is... The real label is The teacher model outputs logits as The student model outputs logits as Knowledge distillation updates the central server model by fusing the soft output distribution of the teacher model with the original supervision signal. The loss function is defined as: ; KL divergence measures the difference in output distribution between the teacher and student models, and is equivalent to cross-entropy loss. Ensure student models match real labels Learning, distillation loss weight .
[0077] Furthermore, the central server model parameters are updated by minimizing distillation loss through optimization algorithms (such as SGD): ; in The learning rate. This process allows the central server model to absorb aggregated parameters while retaining the original supervision information. Based on this knowledge, we can complete iterative parameter optimization.
[0078] S205: Global Model Update Feedback (Model Synchronization Mechanism): The coordinating server feeds back the aggregated global model parameter update values in encrypted form to each local client. Each client decrypts and updates its own model locally according to the feedback update values, completing one round of federated model training.
[0079] S206: Iterative Training and Optimization: Repeat steps S203-S205 multiple times to continuously optimize the lip-reading model. This ensures that the model fully utilizes data from all parties while maintaining data privacy and security, ultimately resulting in a high-performance lip-reading model that can adapt to different data distributions. This iterative process ensures continuous model optimization. The federated learning process provided in this embodiment can be found in the appendix. Figure 4 .
[0080] Secondly, embodiments of the present invention provide a federated lip-reading system, as shown in the appendix. Figure 3 As shown, it includes: The central server is used to initialize the global lip-reading model, distribute the model to local clients, and aggregate encrypted local model parameters to generate updated global model parameters.
[0081] Multiple local clients are used to train the global model locally based on local lip movement data, generate local model parameters, and upload them to the central server through an encryption module.
[0082] An encryption module, integrated into the local client, is used to implement differential privacy and homomorphic encryption on local model parameters; The aggregation module, integrated into the central server, is used to aggregate encrypted local model parameters to generate updated global model parameters.
[0083] The personalization adjustment module, integrated into the local client, is used to fine-tune the global model to adapt it to individual user characteristics.
[0084] The federated lip-reading system provided in this invention utilizes federated learning to distribute and store lip movement data from multiple users across their respective devices, avoiding centralized data collection and transmission and addressing the data scarcity problem. Federated learning updates the model only on the local device; user lip movement data remains locally and is never uploaded to the cloud, ensuring personal privacy. Federated learning only transmits model parameters (usually encrypted), rather than raw data, significantly reducing communication and storage costs, making it suitable for resource-constrained scenarios. By combining local model updates from multiple users, federated learning can aggregate global knowledge, improving the model's generalization ability and recognition accuracy, especially when there are significant differences between user data.
[0085] In this embodiment, the central server and local client include modules for performing the functions described above, which will not be elaborated further in this embodiment.
[0086] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Additionally, the terms "front," "back," "left," "right," "upper," and "lower" in this document refer to the placement shown in the accompanying drawings.
[0087] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A federalized lip-reading method, characterized in that, include: The central server initializes a global lip-reading model and distributes the global model to multiple local clients; Each local client trains the global model locally based on local lip movement data, generates local model parameters, and uploads the parameters to the central server after encrypting them using differential privacy mechanism and homomorphic encryption technology. The central server aggregates the encrypted parameters from all clients and generates updated global model parameters, then feeds back the updated global parameters to each local client. The local client adjusts the local model based on the updated global parameters and fine-tunes the local model through a personalized model adjustment mechanism; Repeat the above steps until the model converges, and finally implement lip reading recognition on the local client.
2. The federalized lip-reading method according to claim 1, characterized in that, The global lip-reading recognition model is a deep learning-based sequence modeling network that employs an end-to-end learning framework based on a connection-time classification loss function to maintain global system labels in order to support lip-reading recognition in different languages.
3. The federalized lip-reading method according to claim 1, characterized in that, The encryption of the parameters using differential privacy mechanism and homomorphic encryption technology further includes: During the local model training phase, noise is added to the gradient or model parameters to implement differential privacy. The uploaded model parameters are encrypted using homomorphic encryption technology to prevent the central server from decrypting and obtaining the original data.
4. The federalized lip-reading method according to claim 1, characterized in that, The central server uses a federated averaging algorithm and model distillation technology to generate updated global model parameters. The federated averaging algorithm weights the aggregation based on the amount of local data on the client.
5. The federalized lip-reading method according to claim 4, characterized in that, The federated averaging algorithm further incorporates a dynamic aggregation mechanism, adjusting weights based on client data quality, model update magnitude, or historical contribution.
6. The federalized lip-reading method according to claim 4, characterized in that, The central server uses a federated averaging algorithm and model distillation techniques to generate updated global model parameters, further including: The central server treats the local models of multiple clients as "teacher models" and generates soft targets through model distillation techniques; The "student model" is trained based on the soft target as the updated global model, and the global parameters are optimized by minimizing the KL divergence loss.
7. The federalized lip-reading method according to claim 1, characterized in that, The personalized model adjustment mechanism includes: The local client fine-tunes the global model based on its own lip movement data to generate a local model that adapts to the user's lip movement characteristics; The fine-tuning process preserves the global knowledge generated by federated learning while adapting to individual differences in lip movement characteristics among users.
8. The federalized lip-reading method according to claim 1, characterized in that, Prior to local training, the client performs data preprocessing, including: The local lip movement video is normalized, noise is filtered, and the lip region is extracted and cropped to generate standardized input.
9. The federalized lip-reading method according to claim 1, characterized in that, The method also includes a dynamic reward mechanism that allocates rewards based on the client's contribution, which is evaluated through model update quality or data volume.
10. A federal-style lip-reading system, characterized in that, include: The central server is used to initialize the global lip reading model, distribute the model to local clients, and aggregate the encrypted local model parameters to generate updated global model parameters. Multiple local clients are used to train the global model locally based on local lip movement data, generate local model parameters, and upload them to the central server through an encryption module. An encryption module, integrated into the local client, is used to implement differential privacy and homomorphic encryption on local model parameters; The aggregation module, integrated into the central server, is used to aggregate encrypted local model parameters to generate updated global model parameters. The personalization adjustment module, integrated into the local client, is used to fine-tune the global model to adapt it to individual user characteristics.