A cross-device biometric multi-metamorphic learning system

Abstract

The application discloses a cross-device biological feature multi-metamorphic learning system, comprising a client device and a global model aggregation server. The client device is equipped with a multi-metamorphic feature encoder for mapping multi-source biological feature data to a unified embedding representation space; a local differential privacy module for adding perturbation to the embedding representation according to the assigned noise intensity; and a local model training engine for updating local parameters. The server comprises a model aggregation engine for aggregating local parameters to generate a global model; a privacy risk perceiver for monitoring bias accumulation and dynamically adjusting the noise budget allocation strategy of each biological modality; and a machine forgetting learning engine for responding to data deletion requests and performing approximate wiping operations. The application effectively solves the problems of unified representation of multi-modal biological feature data in federated learning, dynamic balance of privacy protection and model accuracy, and efficient wiping of data traces, significantly improving the safety, practicality and compliance of the system.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of information technology, and particularly relates to a cross-device biometric multi-metamorphic learning system. BACKGROUND

[0002] As a distributed machine learning technique, federated learning enables data collaboration modeling while protecting user privacy, and has become an important research direction in the field of artificial intelligence. It allows multiple devices or institutions to jointly train models without sharing raw data, and is widely used in finance, healthcare, biometrics, and other fields. However, with the widespread use of multi-modal biometric data (such as faces, fingerprints, irises, etc.), federated learning faces significant challenges in handling complex data and meeting diverse business needs. How to balance privacy protection, model accuracy, and data management requirements has become a core issue in promoting the practical application of this technology. Existing methods often struggle to effectively integrate the features of different types of data when dealing with multi-modal biometric data. Face images, fingerprint textures, and iris details differ greatly in data structure and feature distribution. Traditional methods usually design feature extraction for a single modality, resulting in inconsistent feature representations across modalities. For example, in a biometric identification scenario, a system may need to handle both user face and fingerprint data, but existing methods struggle to map these heterogeneous data to a unified space for processing. This inconsistency in feature representation limits the model's ability to analyze multi-modal data comprehensively. A deeper technical difficulty lies in dynamically adjusting data processing strategies to adapt to the sensitivity differences of different modalities while protecting privacy. Biometric data is highly sensitive and unique, requiring high levels of privacy protection. For example, fingerprint data may be more easily reversed to reveal a user's identity than face data, so stronger privacy protection measures are needed. However, overly strong protection mechanisms (such as adding excessive noise) can significantly reduce model recognition accuracy. For example, in a cross-institution identity verification system, an institution may experience a decrease in fingerprint recognition rate due to excessive privacy protection, affecting user experience. Balancing privacy protection strength and model performance across different modalities is a key technical challenge. In addition, increasing user data management requirements further exacerbates this problem. In actual business, users may request to delete their data to comply with privacy regulations, but existing federated learning systems struggle to accurately assess and eliminate the impact of specific user data on the global model. For example, in a multi-institutional biometric identification system, a user may request to delete their iris data, and the system needs to accurately eliminate the impact of this data without affecting the overall model performance. This dynamic data management requirement intertwines with privacy protection and model accuracy requirements, making the problem even more complex.

[0003] Therefore, how to achieve a unified representation of different modal features in federated learning of multimodal biometric data, while dynamically balancing the strength of privacy protection and model performance, and effectively responding to data deletion requests, has become a key issue in promoting the application of federated learning in the field of biometrics. Summary of the Invention

[0004] In view of this, the present invention provides a cross-device biometric multi-mimicry learning system, mainly comprising:

[0005] One or more client devices are used to collect multi-source biometric data;

[0006] A global model aggregation server is communicatively connected to the one or more client devices;

[0007] The client device includes:

[0008] A multi-mimicry feature encoder is used to receive raw biometric data and map it to a unified embedding representation space;

[0009] The local differential privacy module is used to add noise perturbation to the unified embedding representation output by the multi-mimicry feature encoder;

[0010] A local model training engine is used to update local model parameters using perturbed embedding representations;

[0011] The global model aggregation server includes:

[0012] The model aggregation engine is used to aggregate local model parameters from multiple client devices to generate a global model;

[0013] A privacy risk sensor is used to monitor the accumulation of biases during the aggregation process and dynamically adjust the noise budget allocation strategy for the next round of training based on the global sensitivity of each biological modality.

[0014] The machine forgetting learning engine is used to respond to user data deletion requests, calculate the impact of the data to be deleted on global model parameters, and perform an approximate erasure operation.

[0015] Furthermore, the multi-mimicry feature encoder is a neural network model based on the Transformer architecture, which dynamically calculates and fuses features from different biological modalities through a self-attention mechanism to generate a unified embedding representation that is independent of the modality.

[0016] Furthermore, the privacy risk sensor is configured to perform the following operations:

[0017] Based on the global sensitivity Sm of each biological modality in the multi-source biometric data and the preset privacy budget (ε,δ), the initial noise intensity σ of each modality is calculated according to the standard deviation formula of the Gaussian mechanism. m ,Right now:

[0018] ;

[0019] After each round of global model aggregation, the L2 norm difference between the current global model parameters and the parameters of the previous round is calculated as the cumulative bias value B:

[0020] ;

[0021] If B exceeds the preset threshold B threshold Then, a control signal is generated to adjust the noise intensity of the highly sensitive biological modality as follows:

[0022] ;

[0023] Where α is an adjustable hyperparameter used to control the adjustment range.

[0024] Furthermore, the machine forgetting learning engine employs a second-order optimization algorithm based on Newton's method and introduces an empirical Fisher information matrix to approximate the Hessian matrix to reduce computational complexity, including:

[0025] An approximate solution for calculating the impact Δw of the data to be deleted on the global model parameters:

[0026] ;

[0027] Where H is the empirical Fisher matrix. L(D delete ) represents the gradient on the data to be deleted, obtained by operating w. new =w old Δw updates global model parameters, enabling efficient erasure of data traces.

[0028] Furthermore, the global model aggregation server also includes a multi-objective optimization verification module, used for:

[0029] Receive the new global model after it has been processed by the machine forgetting learning engine;

[0030] Evaluate the recognition accuracy of the new global model on the test dataset;

[0031] The Moments Accountant method, which uses a Gaussian mechanism, calculates the cumulative privacy loss and quantifies the degree of privacy protection based on the remaining privacy budget.

[0032] Verify the output stability of the new global model;

[0033] Based on the preset Pareto optimal strategy, a trade-off is made between recognition accuracy, privacy protection and stability, and the final optimized adaptive noise mechanism parameters are output.

[0034] Furthermore, the multi-source biometric data includes any two or more of the following combinations: face image, fingerprint image, iris image, voiceprint signal, and electrocardiogram signal.

[0035] Furthermore, the local differential privacy module adds noise to the embedded representation using a Gaussian or Laplace mechanism, the magnitude b of which is related to the noise intensity σ assigned by the privacy risk sensor. m Proportional.

[0036] Furthermore, the model aggregation engine uses a federated averaging algorithm for model aggregation. During the aggregation process, based on the weights w_m of each biological modality output by the privacy risk perceiver, a weighted average is performed on the components of the corresponding modality in the client model parameters.

[0037] This invention also provides a cross-device biometric multi-mimicry learning method, comprising:

[0038] Collect multi-source biometric data from multiple clients;

[0039] By deploying multi-mimicry feature encoders on various clients, biometric data from different modalities are mapped to a unified embedding representation space;

[0040] Differential noise perturbation is added to the unified embedding representation based on the noise intensity assigned by the privacy risk perceiver;

[0041] The perturbated embedding representation is used to update the model parameters locally on each client.

[0042] Aggregate local model parameters from multiple clients to generate a global model;

[0043] The system monitors the accumulation of biases during the model aggregation process and dynamically adjusts the noise budget allocation strategy for the next round of training based on the global sensitivity of each biological modality.

[0044] In response to a user's data deletion request, the impact of the data to be deleted on the global model parameters is calculated using a machine forgetting learning algorithm, and an approximate erasure operation is performed.

[0045] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the method.

[0046] The technical solutions provided by the embodiments of the present invention may include the following beneficial effects:

[0047] This invention discloses a cross-device biometric multi-mimetic learning system, addressing the challenge of balancing privacy protection, model accuracy, and data deletion requirements in multimodal biometric data processing. The solution is achieved through a collaborative mechanism of a multi-mimetic feature encoder, a privacy risk sensor, and a machine forgetting learning engine. First, a Transformer-based multi-mimetic feature encoder maps biometric data from different modalities, such as face, fingerprint, and iris scans, to a unified embedding representation space via a self-attention mechanism, ensuring modality-independent feature fusion. Second, a local differential privacy module adds noise to the embedding representation using Gaussian or Laplacian mechanisms, dynamically adjusting the noise intensity in conjunction with the privacy risk sensor to effectively balance privacy protection and model accuracy. The privacy risk sensor dynamically optimizes the noise budget based on global sensitivity and accumulated bias, reducing the weights of highly sensitive modalities to control privacy risks. Furthermore, the machine forgetting learning engine employs a second-order optimization algorithm using Newton's method to accurately calculate and erase the impact of user data on the global model, responding to data deletion requests. A multi-objective optimization verification module further balances recognition accuracy, privacy protection, and model stability, outputting adaptive noise mechanism parameters. This invention achieves efficient processing of multimodal data, strong privacy protection, and flexible data management, significantly improving the security and practicality of federated learning systems. Attached Figure Description

[0048] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0049] Figure 1 This is a schematic diagram of a cross-device biometric multi-mimicry learning system according to the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and thoroughly described below with reference to the accompanying drawings. The described embodiments are merely some embodiments of the present invention.

[0051] Figure 1 A cross-device biometric multi-mimicry learning system is shown, including one or more client devices and a global model aggregation server that communicates with one or more client devices.

[0052] In real-world applications, multiple client devices are distributed across different physical locations, each equipped with corresponding biometric data acquisition hardware. These client devices can be smartphones, tablets, access control systems, time and attendance devices, or dedicated biometric identification terminals. Each client device collects one or more types of biometric data based on its hardware configuration and application scenario.

[0053] Before collecting biometric data, first configure the multi-source biometric data acquisition module.

[0054] The multi-source biometric data acquisition module coordinates the work of different types of sensors to ensure the synchronization and consistency of data acquisition. This module first detects the available biometric acquisition hardware on the current client device, including cameras, fingerprint sensors, iris scanners, microphones, and ECG sensors. For facial image acquisition, the module uses the front or rear camera, setting an appropriate resolution and frame rate, typically 640×480 pixels or higher, to ensure the clarity of facial features. Fingerprint image acquisition is achieved through capacitive or optical fingerprint sensors, with a typical resolution of 500 dpi, capable of capturing detailed fingerprint features including ridges, bifurcation points, and endpoints.

[0055] After collecting biometric data, perform biometric data preprocessing operations.

[0056] Raw biometric data often contains interference factors such as noise, lighting variations, and pose changes, requiring preprocessing to improve the accuracy of subsequent processing. For face images, preprocessing includes face detection, alignment, and normalization. Face detection uses a cascaded classifier or deep learning detector to locate face regions in the image, and then uses keypoint detection algorithms to determine key positions such as the corners of the eyes, the tip of the nose, and the corners of the mouth. Geometric transformations are then performed based on these keypoints to achieve face alignment. The normalization process adjusts the face image to a uniform size and lighting conditions.

[0057] Fingerprint image preprocessing includes image enhancement, binarization, and thinning. Image enhancement removes noise and highlights ridge structures using filtering algorithms; common methods include directional filtering and frequency domain enhancement. Binarization converts a grayscale image to a black and white image, with ridge regions in black and valley regions in white. Thinning reduces the ridge width to a single pixel, facilitating subsequent feature extraction.

[0058] Iris image preprocessing begins with iris localization, using circular Hough transform or active contour modeling to determine the inner and outer boundaries of the iris. The annular iris region is then expanded into a rectangular image to eliminate the effects of pupil size variations and eyelid occlusion. Voiceprint signal preprocessing includes pre-emphasis, framing, windowing, and endpoint detection to remove silent segments and extract valid speech fragments.

[0059] A biometric data quality assessment mechanism has also been established for the collected biometric data.

[0060] Data quality directly impacts the effectiveness of subsequent model training and recognition; therefore, a robust quality assessment mechanism is necessary. This mechanism evaluates the quality of collected data across multiple dimensions, including sharpness, completeness, consistency, and usability. For face images, the quality assessment considers image resolution, illumination uniformity, facial pose angle, expression variations, and occlusion level. The system calculates the Laplacian variance of the image as a sharpness metric, assesses the uniformity of illumination distribution, and detects whether the facial deflection angle is within acceptable limits.

[0061] Fingerprint image quality assessment focuses on ridge sharpness, image contrast, and acquisition area. The system calculates the consistency and continuity of ridge direction, evaluates the image signal-to-noise ratio, and checks whether the effective acquisition area meets the minimum requirements. Iris image quality assessment includes the visible proportion of the iris region, image focus sharpness, and eyelid occlusion. Voiceprint signal quality assessment considers signal-to-noise ratio, speech intelligibility, and effective speech duration.

[0062] The client device includes a multi-mimicry feature encoder, a local differential privacy module, and a local model training engine; the global model aggregation server includes a model aggregation engine, a privacy risk perceiver, and a machine forgetting learning engine.

[0063] By deploying multi-mimicry feature encoders on various clients, raw biometric data from different modalities are mapped to a unified embedding representation space.

[0064] The multi-mimetic feature encoder is one of the core components of the system, responsible for transforming data from different biometric modalities into a unified mathematical representation. Based on the Transformer architecture, this encoder can process sequential feature data and capture the correlations between different modalities through a self-attention mechanism. The encoder's design considers the differences in data characteristics across different biometric modalities, achieving cross-modal feature fusion through a shared representation learning framework.

[0065] In one embodiment, a multi-mimicry feature encoder is used to receive raw biometric data and map it to a unified embedding representation space, specifically including:

[0066] Step S1: Convert biometric data of different modalities into initial feature sequences. For different modalities such as face images and fingerprint images, key feature points are first extracted through preprocessing to form a serialized representation to ensure compatibility of subsequent fusion.

[0067] Step S2: Add modality-specific positional encoding to the initial feature sequence. The positional encoding is a vector sequence generated based on a sine function, used to mark the relative position of each modality in the sequence. For example, for a face image sequence, the added encoding vector will embed modality identifiers to distinguish it from the fingerprint sequence.

[0068] Step S3: Input the sequence with added position encoding into a shared Transformer encoder, wherein the Transformer encoder consists of multiple layers of self-attention layers and a feedforward network, each layer processing the sequence to capture cross-modal dependencies.

[0069] Step S4: The correlation between different modal features is dynamically calculated and fused through the self-attention mechanism in the Transformer encoder to generate a unified embedding representation that is independent of modality. The self-attention mechanism calculates the dot product attention score between the query, key and value vectors and fuses geometric features such as face and texture features of fingerprint to form a fixed-dimensional embedding vector.

[0070] In one embodiment, step S1, which converts biometric data from different modalities into an initial feature sequence, involves using a convolutional neural network to extract edges and textures from face images, forming a vector sequence of length 512. For fingerprint images, a Gabor filter is used to extract ridge directions, forming a sequence of similar length. This conversion ensures data consistency between modalities, facilitates subsequent unified mapping, and improves the accuracy of cross-device biometric identification.

[0071] Adding modality-specific positional encoding in step S2 can prevent modality confusion. For example, when processing a combination of face and voiceprint, the positional encoding vector is like [sin(pos / 10000^(2i / d)),cos(pos / 10000^(2i / d))], where pos is the position, i is the dimension index, and d is the model dimension. This helps the Transformer retain modality uniqueness during fusion, thereby improving the robustness of the embedded representation.

[0072] In one embodiment, the shared Transformer encoder in step S3 is configured with 6 layers, each with 8 attention heads and a dimension of 256. This configuration can effectively handle noisy data input in cross-device scenarios and ensure the stability of the mapping process.

[0073] In step S4, the self-attention mechanism is fused by normalizing the attention weights with softmax and calculating a weighted sum. For example, if the attention scores of the face feature vector and the fingerprint vector are higher than 0.5, the fusion is strengthened to generate a unified 128-dimensional embedding representation. This not only reduces modal heterogeneity but also improves the recognition efficiency under privacy protection.

[0074] In one embodiment, for the overall process of steps S1 to S4, in the combined scenario of iris image and ECG signal, after the initial sequence transformation, position encoding is added. After inputting into Transformer, self-attention captures the temporal correlation between iris texture and ECG waveform, and outputs a unified embedding for subsequent privacy perturbation. This method is beneficial for maintaining data consistency in multi-device federated learning.

[0075] In one embodiment, a local differential privacy module is used to add noise perturbation to the unified embedding representation output by the multi-mimetic feature encoder. Specifically, this includes:

[0076] Step S1: Receive the unified embedding representation generated by the multi-mimicry feature encoder. This representation is a vector sequence that maps multi-source biometric data, such as face images and voiceprint signals, into a unified space.

[0077] Step S2, based on the noise intensity σ assigned by the privacy risk sensor m A Gaussian mechanism is used to add noise perturbation to the unified embedding representation. The Gaussian mechanism refers to injecting random noise following a Gaussian distribution into the data to meet differential privacy requirements. The variance of the noise is related to σ. m It is proportional to ensure that the output distributions of adjacent datasets are difficult to distinguish.

[0078] Step S3: Calculate the perturbation embedding representation after adding noise, and use it as the input for subsequent local training.

[0079] For example, after the client device acquires face and iris images, the multi-mimicry feature encoder outputs a unified embedding representation, and the local differential privacy module, based on σ... m =2.0 Gaussian noise is added, with the noise following a distribution of mean 0 and variance 4.0. After this processing, the value of each dimension of the embedding representation changes from 1.5 to 1.7, avoiding the direct leakage of sensitive biometric features, enhancing user privacy protection, and maintaining the usability of features for model training.

[0080] In one embodiment, for highly sensitive modes such as electrocardiogram signals, σ m Increasing it to 3.0 adds greater noise intensity, and the stability of the embedded representation after perturbation is maintained within 0.5 through variance checks. This provides better control over privacy budgets and reduces the risk of bias accumulation.

[0081] In one embodiment, a local model training engine is used to update local model parameters using perturbed embedding representations. Specifically, this includes:

[0082] Step S1: Using the perturbed embedding representation as input, local training is performed using the stochastic gradient descent algorithm. The stochastic gradient descent algorithm is a process of calculating the gradient of the loss function with respect to the parameters and iteratively updating the parameters with mini-batch data, where the learning rate is 0.01.

[0083] Step S2: Calculate the gradient based on the local dataset and update the local model parameters, such as adjusting the weights from 0.8 to 0.75.

[0084] For example, when training with perturbation embedding representation, the parameters converge after 5 iterations for fingerprint image modalities, and the accuracy is improved to 95%, achieving effective local learning under privacy protection and avoiding the risks brought about by global data sharing.

[0085] In one embodiment, the training engine uses a variant of the Adam optimizer in conjunction with the voiceprint signal. The Adam optimizer accelerates the convergence process by adaptively adjusting the learning rate, with an initial learning rate of 0.001. This results in more stable parameter updates, especially in multimodal scenarios with noise interference.

[0086] In one embodiment, the model aggregation engine is used to aggregate local model parameters from multiple client devices to generate a global model, specifically including:

[0087] Step S1: Receive local model parameters uploaded from various client devices. These parameters are obtained by updating the perturbed embedding representation through the local training engine.

[0088] Step S2: Based on the weights provided by the privacy risk perceiver, the local model parameters from different clients are weighted. The weights are calculated based on the global sensitivity and cumulative bias of each biological modality to ensure that the parameters of high-sensitivity modalities are appropriately adjusted in the aggregation to balance privacy and accuracy.

[0089] Step S3: Apply the federated averaging algorithm to average the weighted parameters and generate the initial global model parameters.

[0090] Step S4: Combine the initial global model parameters with the previous global model and perform a smooth update to form the final global model.

[0091] For example, in a scenario involving biometric data processing of facial images and voiceprint signals, suppose there are three client devices, each collecting data from different modalities, training locally, and then uploading the parameters. The model aggregation engine first collects these parameters; for example, the parameter vector for client 1 is [0.5, 0.3, 0.7], for client 2 it is [0.4, 0.35, 0.65], and for client 3 it is [0.45, 0.32, 0.68]. A privacy risk perceiver calculates weights; if the facial modality is highly sensitive, the weights are 0.4, 0.3, and 0.3. Then, a weighted average is taken to obtain [0.45, 0.325, 0.67] as the global parameter. This aggregation method helps reduce the risk of privacy leakage because the weight adjustment reduces the impact of highly sensitive data while maintaining the model's recognition accuracy above 85%.

[0092] In one possible implementation, step S2 weights the local model parameters based on the weights provided by the privacy risk sensor. The weights w_m are negatively correlated with the cumulative bias value B and the modal sensitivity Sm, ensuring the stability of the aggregation process. For example, if the cumulative bias value B is 0.1 and the sensitivity Sm is 2, the weight is 0.5. The weighted parameters focus more on low-sensitivity modes, improving the model's generalization ability and avoiding overfitting to specific device data.

[0093] For example, step S3 involves applying the federated averaging algorithm, which involves summing all weighted parameters and dividing by the number of clients to generate initial global model parameters. In the face and fingerprint combination scenario, if the averaged parameter vector is [0.46, 0.33, 0.66], this algorithm helps the global model achieve 90% accuracy in cross-device verification, which is beneficial to the overall robustness of the system.

[0094] In one embodiment, the smooth update in step S4 uses an exponential moving average, such as new parameter = 0.9 × old parameter + 0.1 × initial parameter, to ensure model continuity. For example, in consecutive training epochs, this method reduces the global model's loss function from 1.2 to 0.8, which has the beneficial effect of accelerating convergence and improving the stability of biometric recognition.

[0095] In one embodiment, the privacy risk perceiver is used to monitor the accumulation of biases during the aggregation process and dynamically adjust the noise budget allocation strategy for the next round of training based on the global sensitivity of each biological modality, specifically including:

[0096] Step S1: Based on the global sensitivity Sm of each biological modality in the multi-source biometric data and the preset privacy budget (ε, δ), calculate the initial noise intensity σ of each modality. mThe global sensitivity Sm is determined by assessing the potential risk of each biometric modality, such as a facial image or voiceprint signal, in the event of data leakage. For example, facial images have a higher Sm because they are easier to identify individuals. Based on the global sensitivity Sm of each biometric modality in the multi-source biometric data and the preset privacy budget (ε,δ), the initial noise intensity σ of each modality is calculated according to the standard deviation formula of the Gaussian mechanism. m ,Right now:

[0097] ;

[0098] After each round of global model aggregation, the L2 norm difference between the current global model parameters and the parameters of the previous round is calculated as the cumulative bias value B:

[0099] ;

[0100] If B exceeds the preset threshold B threshold Then, a control signal is generated to adjust the noise intensity of the highly sensitive biological modality as follows:

[0101] ;

[0102] Where α is an adjustable hyperparameter used to control the adjustment range.

[0103] Step S11: Calculate the initial noise intensity σ for each biological modality based on the global sensitivity Sm and the privacy budget (ε,δ). m This ensures that noise levels are proportional to sensitivity, forming an initial layer of privacy protection.

[0104] Step S12, after calculation, the initial noise intensity σ m The noise is allocated to the local differential privacy module for perturbation of the subsequent embedded representation, so that the initial noise intensity serves as input to guide the noise addition process.

[0105] Step S2: After each round of global model aggregation, the change in model weights or loss function is calculated as the cumulative bias value B. The variance calculation process involves comparing the aggregated model parameters with the parameters of the previous round and calculating the average of the squared differences of each parameter dimension. For example, the variance function result is calculated for the model weight vector as B, ensuring that the degree of bias accumulation is monitored.

[0106] Step S21: If the cumulative deviation value B exceeds a preset threshold, a control signal is generated to increase the noise intensity σ corresponding to the highly sensitive biological modality. m Alternatively, its weight w_m in model aggregation can be reduced, where the preset threshold, such as 0.05, is set experimentally to determine whether the bias affects model stability. Increasing σ... m The process is to set the current σ mMultiplying by a factor such as 1.2, while reducing w_m is equivalent to dividing the weight by the same factor, ensures that the privacy of highly sensitive modalities such as iris images is enhanced.

[0107] Step S22: Based on the generated control signal, update the noise budget allocation strategy for the next round of training. In this way, the control signal, as an input, directly affects the dynamic changes in noise intensity, forming a closed-loop adjustment.

[0108] In one possible implementation, the initial noise intensity σ is calculated in step S1. m For different combinations of biological modalities, such as face images and fingerprint images, assuming that Sm for faces is 0.8, Sm for fingerprints is 0.6, ε is 1.0, δ is a preset standard value, and σ m The values ​​are 0.8 and 0.6 respectively. This calculation has the beneficial effect of providing more noise protection for highly sensitive modes, reducing the risk of privacy leaks, while maintaining the overall accuracy of the model.

[0109] For example, in step S2, if the variance of the aggregated model's prediction of the test data is 0.07, which exceeds the threshold of 0.05, an adjustment is triggered. This helps to detect model instability caused by the accumulation of bias as early as possible and is beneficial to improving the long-term privacy protection performance of the system.

[0110] In one embodiment, after the control signal is generated in step S21, if σ is increased... m For the voiceprint signal mode, the noise intensity is increased from 0.5 to 0.6, which not only enhances privacy, but also balances accuracy by reducing the aggregation weight w_m from 0.4 to 0.3, achieving a dynamic trade-off between privacy and performance.

[0111] For example, consider a scenario where the system processes a combination of ECG signals and iris images. When the cumulative deviation B reaches 0.06, the control signal preferentially increases the σ of the iris. m Because its Sm is high at 0.9, it ensures targeted protection of sensitive data. At the same time, the noise budget in subsequent rounds is adjusted based on this, forming a complete chain from monitoring to optimization, which is beneficial to the robustness of cross-device biometric learning.

[0112] In one embodiment, a machine forgetting learning engine is configured to, in response to a user data deletion request, calculate the impact of the data to be deleted on global model parameters and perform an approximate erasure operation. Specifically:

[0113] Step S1, in response to the user data deletion request.

[0114] In one embodiment, upon receiving a data deletion request submitted by a user through a client device, the system immediately triggers the engine activation process. This process ensures that the request is verified as legitimate, thereby initiating subsequent calculations.

[0115] Step S2: Calculate the impact of the data to be deleted on the global model parameters.

[0116] In one embodiment, the machine forgetting learning engine employs a second-order optimization algorithm based on Newton's method and introduces an empirical Fisher information matrix to approximate the Hessian matrix to reduce computational complexity, including:

[0117] An approximate solution for calculating the impact Δw of the data to be deleted on the global model parameters:

[0118] ;

[0119] Where H is the empirical Fisher matrix. L(D delete ) represents the gradient on the data to be deleted.

[0120] For example, when processing biometric data containing iris images, assuming the data to be deleted is an iris scan sequence of a specific user, the calculation process will evaluate the impact of this data on the perturbation of the model embedding space and generate a vector adjustment with a Δw value of approximately 0.05 to maintain the overall model stability.

[0121] In one embodiment, for ECG signal modes, the computation is extended to a dynamic noise budget, and the Hessian approximation accuracy is adjusted based on the output of a privacy risk sensor to avoid bias accumulation. This extension improves applicability across device environments, delivers better forgetting efficiency, and helps protect user privacy without sacrificing model performance.

[0122] Step S3: Perform an approximate erasure operation.

[0123] In one embodiment, based on the calculated Δw, the operation w is performed. new =w old Δw updates the global model parameters, achieving efficient erasure of data traces. Specifically, this operation is performed on the aggregation server, ensuring that the new model's recognition accuracy on the test dataset decreases by no more than 2%, while the privacy budget ε remains within a preset range. This erasure operation is particularly effective in federated learning frameworks because it allows for rapid responses to deletion requests without retraining the entire model.

[0124] For example, in one scenario, a user requests the deletion of their fingerprint image data. After erasure, the output stability of the new model is verified by a variance check to be below 0.01, proving the effectiveness of the operation and contributing to enhanced system compliance and user trust.

[0125] In one embodiment, for multimodal combinations such as facial images and voiceprint signals, the erasure operation further integrates multi-objective optimization verification, assessing the failure probability δ of the new model to be 10^-5, ensuring stability, and outputting adaptive noise parameters. This approach yields significant results in highly sensitive biological modalities, such as reducing potential leakage risk by up to 30%.

[0126] The above implementation methods and embodiments closely adhere to the core functions of the machine forgetting learning engine, ensuring efficient privacy protection by accurately calculating the impact and executing erasure operations when responding to user data deletion requests.

[0127] The global model aggregation server also includes a multi-objective optimization verification module.

[0128] In one embodiment, step S1, the multi-objective optimization verification module receives a new global model processed by the machine forgetting learning engine, specifically including step S11, obtaining a new global model with updated parameters from the machine forgetting learning engine through a communication interface, wherein the parameters include weights and bias values, which have been adjusted after the erasure operation to remove the influence of specific data.

[0129] Step S12: Load the received model into the validation environment to ensure that the model structure is consistent with the original global model so that the model can be used for prediction in subsequent evaluations.

[0130] Step S2 evaluates the recognition accuracy of the new global model on the test dataset, specifically including step S21, extracting samples from the pre-stored test dataset, which contain a variety of biometric features such as face and voiceprint combinations.

[0131] Step S22: Use the new global model to perform forward propagation on the samples and calculate the matching rate between the predicted label and the real label as an indicator of recognition accuracy. For example, a matching rate of 95% indicates high accuracy.

[0132] The above evaluation process helps to confirm that the model still retains its effective recognition ability after forgetting operations, avoiding the risk of system unavailability caused by a significant decrease in accuracy.

[0133] Step S3, evaluate the privacy protection of the new global model, specifically including:

[0134] Step S31: Calculate the cumulative privacy loss (ε) from the start of training to the current round using the Gaussian mechanism's Moments Accountant method. consumed This method provides a tight upper bound on privacy loss by analyzing the moment generation function of the noise addition mechanism;

[0135] Step S32, based on the system's preset total privacy budget (ε) total ,δ), calculate the remaining privacy budget and privacy protection level:

[0136] ;

[0137] Step S33: Conduct a risk assessment based on the Privacy Protection Ratio (PP) value.

[0138] When PP ≥ 0.8, the privacy protection level is "strong".

[0139] When 0.6 ≤ PP < 0.8, the privacy protection level is "Medium".

[0140] When PP < 0.6, the privacy protection level is "weak," requiring noise policy adjustment to be triggered. The system will automatically adjust the noise intensity for the next round. , where γ is the adjustment coefficient, with a default value of 0.5.

[0141] For example, in biometric systems, this computation ensures that the privacy of user data, such as iris images, is not compromised. By precisely quantifying the budget, the system can balance protection and performance, which helps prevent data reverse engineering attacks.

[0142] In one embodiment, verifying the output stability of the new global model in step S4 specifically includes step S41, repeatedly running the model for multiple test inputs and collecting the output vector.

[0143] Step S42: Calculate the variance of the output vector as a stability index. The variance calculation process is to average the values ​​of each dimension and then average the squared differences between each value and the average. If the variance is lower than a preset threshold such as 0.01, it is considered stable.

[0144] Step S43: Compare the variance changes of the model before and after forgetting to ensure that the change does not exceed 10%, in order to confirm that the erasure operation did not introduce unstable factors.

[0145] For example, this verification can detect output fluctuations caused by accumulated biases early on, which is beneficial in maintaining consistent recognition results and avoiding misidentification of users due to noise disturbances in cross-device biometric learning.

[0146] In one embodiment, step S5 involves balancing recognition accuracy, privacy protection, and stability according to a preset Pareto optimal strategy, and outputting the final optimized adaptive noise mechanism parameters. Specifically, step S51 involves constructing a three-dimensional target space, using recognition accuracy, privacy protection, and stability as coordinate axes. The Pareto optimal strategy refers to finding a set of parameters that improves one target without worsening other targets. This is achieved by enumerating the noise intensity σ. m Possible values ​​for the value are such as 0.1 to 1.0, and the score for each combination is calculated.

[0147] Step S52: Quantify the trade-offs using a weighted summation method, for example, assigning a weight of 0.5 to precision, 0.3 to protection, and 0.2 to stability, and calculate σ to maximize the total score. m value.

[0148] Step S53: Output the optimized parameters, such as the adjusted σ. m =0.5, used for noise addition in the next round of training.

[0149] For example, in one possible implementation, for a biometric modality combining face and fingerprint, initial evaluation shows 90% accuracy, 85% protection, and 92% stability, with σ adjusted using a Pareto strategy. m Afterwards, the accuracy slightly decreased to 88%, but the protection level increased to 90%, and the stability remained at 91%. This trade-off is beneficial to the overall system robustness.

[0150] In one embodiment, for highly sensitive modalities such as iris images, the Pareto strategy prioritizes increasing the protection level, outputting σ. m =0.8, which causes the accuracy to drop from 92% to 89%, but the protection level increases from 80% to 95% and the stability increases from 90% to 93%. This adjustment sacrifices a small amount of accuracy for strong privacy, which is beneficial in preventing the leakage of sensitive data.

[0151] For example, it is understandable that in the voiceprint signal scenario, the initial stability is as low as 85%, and the strategy reduces σ m The stability is improved to 92% by 0.3, while maintaining accuracy of 91% and protection of 88%. The multi-objective trade-off ensures that the system can adapt to different modal requirements in dynamic environments and enhance user trust.

[0152] In one embodiment, for cross-device learning of ECG signals, Pareto optimality is achieved by iteratively optimizing the output σ. m =0.6, after balancing, the accuracy reaches 93%, the protection degree 92%, and the stability 94%. This method is beneficial to reduce the accumulation of bias and achieve sustainable model updates.

[0153] The local differential privacy module adds noise to the embedded representation using either a Gaussian or Laplace mechanism. The magnitude b of the added noise is related to the noise intensity σ assigned by the privacy risk sensor. m Proportional.

[0154] In one embodiment, the process of adding noise to the embedded representation by the local differential privacy module specifically includes the following steps.

[0155] Step S1: Choose either the Gaussian or Laplace mechanism as the tool for adding noise. The Gaussian mechanism achieves differential privacy by adding random noise that follows a Gaussian distribution to the data, while the Laplace mechanism adds noise that follows a Laplace distribution. Both can protect privacy without significantly reducing data utility.

[0156] Step S11: Based on the sensitivity of the current biological modality, obtain the noise intensity σ from the privacy risk sensor. m , the σ m It is calculated based on the global sensitivity Sm and the privacy budget (ε, δ), ensuring that the noise level matches the data sensitivity.

[0157] Step S12, determine the noise scale b, making it consistent with σ. m Proportional, for example, b = k•σ m Where k is a proportional constant, the addition of noise can dynamically respond to changes in risk through this proportional relationship, avoiding excessive or insufficient privacy protection.

[0158] Step S2: Add noise generated by the selected mechanism to the unified embedding representation. The unified embedding representation is the result of the multi-mimicry feature encoder mapping the original biometric data to a shared space, such as a vector fused from a face image and a voiceprint signal.

[0159] Step S21: For the Gaussian mechanism, generate a noise vector with a mean of 0 and a standard deviation of b, and add it element by element to the embedding representation; for the Laplace mechanism, generate a noise vector with a scale parameter of b and add it, ensuring that the perturbation of the embedding representation after addition satisfies the differential privacy definition, that is, the difference in output distribution between any two adjacent datasets is controlled by the privacy budget.

[0160] In one embodiment, when selecting a mechanism in step S1, if the biometric data involves highly sensitive modalities such as iris images, the Gaussian mechanism can be preferred because its smooth perturbation on continuous data can better maintain the recognition accuracy of features, while the Laplace mechanism is suitable for discrete data to provide stronger privacy guarantees.

[0161] For example, when processing embedded representations that combine facial and electrocardiogram signals, the privacy risk perceiver assigns σ. m =0.5, then b=1.0 (assuming k=2). After adding noise, each component of the embedding vector is perturbed from [0.8, 0.3] to [0.9, 0.4]. This can effectively blur the individual identity while retaining enough information for local model training, which is beneficial to improving the overall privacy protection of the system without significantly reducing the accuracy of the aggregated model.

[0162] In one embodiment, the noise addition in step S2 can be extended to multi-round training scenarios, with each round based on the updated σ.m Adjust b to ensure that the cumulative privacy budget does not exceed the preset threshold (ε, δ).

[0163] For example, the initial round σ m =0.4, b=0.8; if the cumulative deviation B exceeds the threshold, σ will be increased in the next round. m When b increases to 0.6, b increases to 1.2 accordingly. This dynamic adjustment reduces the impact of accumulated bias on the global model, resulting in a more stable learning process. It can balance privacy and performance, especially when dealing with multiple biometrics in cross-device environments.

[0164] In one embodiment, for noise scale b and σ m The proportional relationship can be adjusted by introducing an adaptive k value, which can be adjusted according to the modality type, such as k=1.5 for voiceprint signals and k=2.0 for face images, to optimize the privacy-utility trade-off for different data.

[0165] For example, in one scenario, the embedding is represented as a 512-dimensional vector, σ m With b=1.0 and b=2.0, the vector variance increases but the mean remains stable after adding Gaussian noise. This helps prevent attackers from inferring the original data from the aggregation model, enhances the system's resistance to attacks, and achieves efficient privacy budget allocation through this proportional mechanism.

[0166] The model aggregation engine uses a federated averaging algorithm to aggregate models, and during the aggregation process, it uses a weighted average of model parameters from different clients based on the weight w_m output by the privacy risk perceiver.

[0167] In one embodiment, the operation process of the model aggregation engine specifically includes the following steps.

[0168] Step S1: Use the federated averaging algorithm to aggregate the models.

[0169] For example, this step updates the global model by collecting the local model parameters uploaded by each client and calculating their average value, ensuring consistency in distributed training.

[0170] In one embodiment, step S2 involves using a weighted average of model parameters from different clients during the aggregation process, based on the weight w_m output by the privacy risk perceiver.

[0171] Specifically, this step first obtains the weight w_m of each biometric modality from the privacy risk perceiver. These weights reflect the sensitivity of the modality. For example, the face image modality has a higher weight to enhance privacy protection. Then, these weights are applied in the federated averaging algorithm for weighted calculation, ultimately generating a more privacy-focused global model.

[0172] Step S21: Determine the contribution of each client model parameter based on the weight w_m output by the privacy risk perceiver.

[0173] For example, when processing multi-source biometric data, if the sensor detects a high cumulative deviation of fingerprint image modalities, it outputs a low w_m value, such as 0.6, thereby reducing the impact of that modality in aggregation and avoiding the risk of privacy leakage.

[0174] Step S22: Apply the weight w_m to the federated average algorithm, perform a weighted summation of the model parameters and divide by the total weight to obtain the weighted average result.

[0175] In one embodiment, this step ensures that model parameters of highly sensitive modalities, such as iris images, participate in aggregation with a smaller weight. For example, when w_m is 0.4, its parameters contribute only 40% of the average value. This helps to balance accuracy and privacy protection and reduces model instability caused by accumulated bias.

[0176] For example, suppose there are three clients that process face, fingerprint and voiceprint data respectively, and the perceptron output w_m is 0.8, 0.5 and 0.7 respectively. When aggregating, each parameter is multiplied by its corresponding w_m and summed, and then divided by the total weight of 2.0 to generate a global model. This method improves the robustness of the system in cross-device biometric learning.

[0177] In one embodiment, for ECG signal modes, if the sensor detects a high deviation, w_m is dynamically adjusted to 0.3. The recognition accuracy of the aggregated model remains above 95%, while the privacy budget consumption is reduced by 10%, demonstrating the beneficial effect of weighted averaging in privacy-sensitive scenarios.

[0178] For example, in another scenario, a mixed modality combining iris and voiceprint is used, with weights w_m set to 0.9 and 0.4. The aggregation process reduces noise interference through weighted averaging, ensuring the stability of the global model after deleting user data and improving the forgetting learning efficiency of the overall system.

[0179] This invention also provides a cross-device biometric multi-mimicry learning method, comprising:

[0180] Collect multi-source biometric data from multiple clients;

[0181] By deploying multi-mimicry feature encoders on various clients, biometric data from different modalities are mapped to a unified embedding representation space;

[0182] Differential noise perturbation is added to the unified embedding representation based on the noise intensity assigned by the privacy risk perceiver;

[0183] The perturbated embedding representation is used to update the model parameters locally on each client.

[0184] Aggregate local model parameters from multiple clients to generate a global model;

[0185] The system monitors the accumulation of biases during the model aggregation process and dynamically adjusts the noise budget allocation strategy for the next round of training based on the global sensitivity of each biological modality.

[0186] In response to a user's data deletion request, the impact of the data to be deleted on the global model parameters is calculated using a machine forgetting learning algorithm, and an approximate erasure operation is performed.

[0187] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the method.

[0188] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A cross-device biometric multi-mimicry learning system, characterized in that, The system includes: One or more client devices are used to collect multi-source biometric data; A global model aggregation server is communicatively connected to the one or more client devices; The client device includes: A multi-mimicry feature encoder is used to receive raw biometric data and map it to a unified embedding representation space; The local differential privacy module is used to add noise perturbation to the unified embedding representation output by the multi-mimicry feature encoder; A local model training engine is used to update local model parameters using perturbed embedding representations; The global model aggregation server includes: The model aggregation engine is used to aggregate local model parameters from multiple client devices to generate a global model; A privacy risk sensor is used to monitor the accumulation of biases during the aggregation process and dynamically adjust the noise budget allocation strategy for the next round of training based on the global sensitivity of each biological modality. The machine forgetting learning engine is used to respond to user data deletion requests, calculate the impact of the data to be deleted on global model parameters, and perform an approximate erasure operation; The privacy risk sensor is configured to perform the following operations: Based on the global sensitivity Sm of each biological modality in the multi-source biometric data and the preset privacy budget (ε, δ), where ε is the upper bound of privacy loss and δ is the failure probability, the initial noise intensity σ of each modality is calculated according to the standard deviation formula of the Gaussian mechanism. m ,Right now: After each round of global model aggregation, the current global model parameters w are calculated. t Compared with the previous round parameter w t-1 The L2 norm difference is used as the cumulative bias value B: If B exceeds the preset threshold B threshold Then, a control signal is generated to adjust the noise intensity of the highly sensitive biological modality as follows: in The noise level before adjustment. The adjusted noise intensity is represented by α, which is an adjustable hyperparameter used to control the adjustment range.

2. The system as described in claim 1, characterized in that, The multi-mimicry feature encoder is a neural network model based on the Transformer architecture. It dynamically calculates and fuses features from different biological modalities through a self-attention mechanism to generate a unified embedding representation that is independent of the modality.

3. The system as described in claim 1, characterized in that, The machine forgetting learning engine employs a second-order optimization algorithm based on Newton's method and introduces an empirical Fisher information matrix to approximate the Hessian matrix to reduce computational complexity, including: An approximate solution for calculating the impact Δw of the data to be deleted on the global model parameters: Where H is the empirical Fisher matrix. L(D delete ) represents the gradient on the data to be deleted, obtained by operating w. new =w old Δw updates the global model parameters, achieving efficient erasure of data traces, where w old The global model parameters before the update, w new These are the updated global model parameters.

4. The system as described in claim 1, characterized in that, The global model aggregation server also includes a multi-objective optimization verification module, used for: Receive the new global model after it has been processed by the machine forgetting learning engine; Evaluate the recognition accuracy of the new global model on the test dataset; The Moments Accountant method, which uses a Gaussian mechanism, calculates the cumulative privacy loss and quantifies the degree of privacy protection based on the remaining privacy budget. Verify the output stability of the new global model; Based on the preset Pareto optimal strategy, a trade-off is made between recognition accuracy, privacy protection and stability, and the final optimized adaptive noise mechanism parameters are output.

5. The system as described in claim 1, characterized in that, The multi-source biometric data includes any two or more of the following combinations: Facial images, fingerprint images, iris images, voiceprint signals, and electrocardiogram signals.

6. The system as described in claim 1, characterized in that, The local differential privacy module adds noise to the embedded representation using a Gaussian or Laplace mechanism, where the magnitude b of the added noise is related to the noise intensity σ assigned by the privacy risk sensor. m Proportional.

7. The system as described in claim 1, characterized in that, The model aggregation engine uses a federated averaging algorithm for model aggregation. During the aggregation process, based on the weights w_m of each biological modality output by the privacy risk perceiver, a weighted average is performed on the components of the corresponding modality in the client model parameters.

8. A cross-device biometric multi-mimicry learning method, applied to the system as described in any one of claims 1-7, characterized in that, The method includes: Collect multi-source biometric data from multiple clients; By deploying multi-mimicry feature encoders on various clients, biometric data from different modalities are mapped to a unified embedding representation space; Differential noise perturbation is added to the unified embedding representation based on the noise intensity assigned by the privacy risk perceiver; The perturbated embedding representation is used to update the model parameters locally on each client. Aggregate local model parameters from multiple clients to generate a global model; monitor the accumulation of biases during the model aggregation process and dynamically adjust the noise budget allocation strategy for the next round of training based on the global sensitivity of each biological modality; In response to a user's data deletion request, the impact of the data to be deleted on the global model parameters is calculated using a machine forgetting learning algorithm, and an approximate erasure operation is performed.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in claim 8.

Images