Speech recognition model training method and electronic equipment
By employing adaptive noise addition and sensitive gradient protection, the privacy leakage problem in the training process of speech recognition models in federated learning is solved, achieving a balance between privacy protection and training performance of speech recognition models.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-19
- Publication Date
- 2026-03-13
AI Technical Summary
In federated learning, how can we balance privacy protection and model training effectiveness during the training process of speech recognition models, and avoid privacy leaks caused by the semantic and acoustic features of speech data during data exchange?
The speech representation is noise-added by adaptively calculating the noise amplitude. Combined with privacy protection processing of sensitive gradient, the model parameters of the speech recognition model are updated. The adaptive noise-adding strategy implements privacy protection in both acoustic and semantic dimensions, and privacy protection is carried out simultaneously at the gradient level.
This approach effectively protects speech privacy during speech recognition model training while maintaining model training effectiveness, avoiding excessive noise interference with feature quality, and ensuring a balance between privacy and model training performance.
Smart Images
Figure CN121662031A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, specifically to a speech recognition model training method and electronic device. Background Technology
[0002] Speech recognition technology refers to the technology of automatically converting audio signals into text, and it is the foundation for realizing human-computer voice interaction. Speech recognition models typically require large-scale speech data for training. Since speech data is distributed among different vendors, federated learning is usually used to obtain speech recognition models.
[0003] However, federated learning requires data exchange between participants and service providers. Since speech data carries both semantic and acoustic features, privacy leaks are likely to occur during the data exchange process. Therefore, there is an urgent need for a way to balance privacy protection and model training effectiveness during the training of speech recognition models. Summary of the Invention
[0004] To address the aforementioned problems, the present invention aims to provide a speech recognition model training method and electronic device that can perform privacy protection processing in both semantic and acoustic dimensions, thus balancing privacy protection and training effectiveness in speech recognition model training.
[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution: On one hand, embodiments of the present invention provide a speech recognition model training method, including: A preset model is used to obtain the speech representation corresponding to the sample speech data, and the speech representation includes acoustic features and semantic features. Based on the semantically important parameters and acoustically unique parameters of the sample speech data, the noise amplitude of the sample speech data is calculated. The speech representation is noise-added based on the noise amplitude to obtain a privacy speech representation. Based on the backpropagation gradient sent by the service provider, the acoustic gradient and semantic gradient are calculated. The backpropagation gradient is calculated by the service provider based on the labeled data and the privacy speech representation sent by all participants. Privacy protection processing is performed on the sensitive gradients in the acoustic gradient and semantic gradient to obtain the privacy acoustic gradient and privacy semantic gradient. The model parameters of the preset model are updated using the privacy acoustic gradient and the privacy semantic gradient to obtain a speech recognition model.
[0006] On the other hand, embodiments of the present invention also provide a speech recognition model training device, comprising: The representation extraction module is used to obtain the speech representation corresponding to the sample speech data using a preset model. The speech representation includes acoustic features and semantic features. An amplitude calculation module is used to calculate the noise amplitude of the sample speech data based on the semantically important parameters and acoustically unique parameters of the sample speech data. A noise-adding module is used to coarsely add noise to the speech representation based on the noise amplitude to obtain a privacy speech representation. The gradient calculation module is used to calculate the acoustic gradient and semantic gradient based on the backpropagation gradient sent by the service provider. The backpropagation gradient is calculated by the service provider based on the label data and the privacy speech representation sent by all participants. A privacy processing module is used to perform privacy protection processing on the sensitive gradients in the acoustic gradient and semantic gradient to obtain a privacy acoustic gradient and a privacy semantic gradient. The parameter update module is used to update the model parameters of the preset model using the privacy acoustic gradient and the privacy semantic gradient to obtain a speech recognition model.
[0007] On the other hand, embodiments of the present invention also provide an electronic device, including a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute steps in any of the speech recognition model training methods provided in embodiments of the present invention.
[0008] On the other hand, embodiments of the present invention also provide a computer-readable storage medium storing a plurality of instructions adapted for loading by a processor to execute steps in any of the speech recognition model training methods provided in embodiments of the present invention.
[0009] On the other hand, embodiments of the present invention also provide a computer program product, including a computer program / instruction, wherein when the computer program / instruction is executed by a processor, it implements the steps in any of the speech recognition model training methods provided in embodiments of the present invention.
[0010] The beneficial effects of the technical solution provided by this invention include at least the following: In this embodiment of the invention, acoustic and semantic features corresponding to sample speech data are obtained using a preset model. Noise amplitude is adaptively calculated based on the semantically important parameters and unique acoustic parameters of the sample speech data, and noise is added to the speech representation based on the noise amplitude to adaptively protect the privacy of the representation. Acoustic and semantic gradients are calculated based on the backpropagation gradient sent by the service provider, and sensitive gradients are identified and processed for privacy protection. The model parameters are then updated using the privacy-protected gradients. Implementing an adaptive noise-adding strategy in both acoustic and semantic dimensions effectively protects acoustic and semantic privacy through noise injection while precisely controlling noise intensity to avoid excessive interference with feature quality. Simultaneously, privacy protection processing is carried out at the gradient level, ultimately achieving a balance between privacy security and the training effect of the speech recognition model. Attached Figure Description
[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0012] Figure 1 This is a schematic diagram illustrating an application scenario of the speech recognition model training method provided in this embodiment of the invention; Figure 2 This is a flowchart illustrating the speech recognition model training method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of data interaction between the participants and the service provider provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of the speech recognition model training device provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation
[0013] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0014] It is understood that, in specific embodiments of the present invention, data involving user voice and related information requires user permission or consent, and the collection, use and processing of such data must comply with the relevant laws, regulations and standards of the relevant countries and regions.
[0015] See also Figure 1 This diagram illustrates an application scenario for a speech recognition model training method. The scenario includes participant 101 and service provider 102, who exchange data via a network. Participant 101 has a bottom model deployed on it, while service provider 102 has a top model deployed on it. Participant 101 uses a pre-defined model to obtain speech representations corresponding to sample speech data. These representations include acoustic and semantic features. Based on semantically important parameters and unique acoustic parameters, it calculates the noise amplitude of the sample speech data. Noise is added to the speech representation based on the noise amplitude to obtain a privacy-preserving speech representation. Participant 101 then sends this privacy-preserving speech representation to service provider 102. Service provider 102 can then calculate the backpropagation gradient based on the labeled data and the privacy-preserving speech representations sent by all participants, and then send the backpropagation gradient back to participant 101.
[0016] Participant 101 can calculate the acoustic gradient and semantic gradient based on the backpropagation gradient; perform privacy protection processing on the sensitive gradients in the acoustic gradient and semantic gradient to obtain the privacy acoustic gradient and privacy semantic gradient; finally, use the privacy acoustic gradient and privacy semantic gradient to update the model parameters to obtain the speech recognition model.
[0017] This embodiment provides a speech recognition model training method applicable to participating parties. It should be noted that this embodiment uses federated learning to obtain the speech recognition model. Federated learning involves multiple parties collaboratively training one or more machine learning models. Figure 2 As shown, the specific process of training this speech recognition model can be as follows: S110. Obtain the speech representation corresponding to the sample speech data using a preset model.
[0018] A pre-set model refers to a model used for speech recognition. A pre-set model is an untrained model that can process speech data but cannot guarantee recognition accuracy. The embodiment of this invention trains the pre-set model to obtain a speech recognition model, which can accurately recognize speech. The pre-set model may include convolutional layers, attention layers, feedforward network layers, and an output layer. Sample speech data is passed sequentially through the convolutional layer, attention layer, feedforward network layer, and output layer.
[0019] The pre-defined model can be split into a bottom model and a top model. The bottom model includes convolutional layers and attention layers, while the top model includes feedforward network layers and an output layer. The bottom model can be deployed to the participants, and the top model can be deployed to the service provider. The participants hold sample speech data, and the service provider holds corresponding label data. The participants convert the sample speech data into a speech representation using the bottom model. This speech representation can include acoustic features and semantic features. Acoustic features refer to features in the sample speech data related to voiceprint, timbre, accent, etc., while semantic features refer to features in the sample speech data related to semantics.
[0020] The bottom model deployed on the participant includes convolutional layers and attention layers. The participant can preprocess the collected speech data to convert the raw speech data into structured features. Optionally, the preprocessing of the sample speech data can involve compensating for the natural attenuation of high-frequency signals in the sample speech data based on filters to obtain compensated speech; performing frame-by-frame windowing on the compensated speech to obtain framed data; then performing Fourier transform on the framed data and passing it through a Mel filter to obtain Mel frequencies; finally, performing discrete cosine transform on the Mel frequencies to obtain Mel spectral features. After obtaining the Mel spectral features, these features can be input into the bottom model to obtain the speech representation corresponding to the sample speech data.
[0021] After the Mel-frequency spectral features are input into the bottom model, they are first fed into a convolutional layer. This convolutional layer consists of three layers: depthwise separable convolution, batch normalization, and activation functions. After processing by these three layers, the frequency features are converted into acoustic features. The acoustic features have a dimension of B. T The dataset is 64-dimensional, with B representing the batch size and T representing the number of frames. The activation function used is the swish activation function.
[0022] The attention layer employs an 8-head self-attention mechanism, and acoustic features are processed by the attention layer to obtain semantic features. Among these, the semantic features are B. T 512 dimensions. Then, the acoustic and semantic features are concatenated according to channels to obtain the speech representation, with a dimension of B. T 576 dimensions.
[0023] S120. Calculate the noise amplitude of the sample speech data based on the semantically important parameters and acoustically unique parameters of the sample speech data.
[0024] After obtaining the speech representation, following the federated learning process, the speech representation needs to be sent to the service provider for forward learning on the top-level model deployed on the service provider. Since the speech representation carries both semantic and acoustic features, noise injection is necessary to prevent privacy breaches.
[0025] For different sample speech data, since they carry different private content, if noise injection with the same noise amplitude is used, it may result in insufficient noise injection for some sample speech data or excessive noise injection.
[0026] Therefore, for different sample speech data, the noise amplitude corresponding to the noise to be injected can be adaptively calculated based on its semantic importance parameters and acoustic uniqueness parameters. Among them, the semantic importance parameters can be used to measure the semantic importance of the sample speech data; the acoustic uniqueness parameters can be used to measure the uniqueness of the acoustic features.
[0027] In some implementations, adaptively calculating the noise amplitude corresponding to the sample speech data may involve matching the sample speech data with each term in a preset terminology library to calculate the semantic importance parameter of the sample speech data; matching the sample speech data with each speech in a preset speech library based on the acoustic features of the sample speech data to calculate the acoustic unique parameter of the sample speech data; and calculating the noise amplitude of the sample speech data based on the semantic importance parameter, the acoustic unique parameter, and the reference amplitude.
[0028] The pre-set terminology library contains multiple terms, each related to the specific scenario in which the speech recognition model will be applied. For example, if the speech recognition model is applied to a medical consultation scenario, the terms in the pre-set terminology library would be common words used in medical consultations. Similarly, if the speech recognition model is applied to an industrial manufacturing scenario, the terms in the pre-set terminology library would be common words used in industrial manufacturing. This pre-set terminology library can be configured in advance and retrieved directly when needed.
[0029] By matching sample speech data with each term in a preset terminology library, the number of terms matched by the sample speech data can be determined. The more terms matched, the more relevant the speech data is to the scenario and the more semantically important it is. Optionally, when calculating the semantic importance parameters of the sample speech data, the sample speech data can be converted into sample speech text; the sample speech text can be matched with each term in the preset terminology library to obtain matched terms; the term matching degree can be calculated based on the number of matched terms and the total number of words in the sample speech text; and the semantic importance parameters of the sample speech data can be calculated using the term matching degree and the term weight of each term.
[0030] To improve terminology matching, sample speech data can be converted into sample speech-text, which can be done using existing speech-to-text models. When matching the sample speech-text with each term, it can be done by detecting whether the sample speech-text contains the term; if it does, the sample speech-text and the term are considered a match, and the term is identified as the matched term. After traversing all terms in a preset terminology database, all matching terms can be identified. The terminology matching degree is calculated by counting the number of matching terms and the total number of words in the sample speech-text, and dividing the number of matching terms by the total number of times. For example, if a medical consultation text contains the terms "hypertension" and "nifedipine," and the total number of words is 10, then the terminology matching degree is 0.2.
[0031] In some embodiments, each term in the preset terminology library can also be assigned a corresponding term weight. In certain scenarios, the term weight can be used to characterize the urgency of the term; the higher the term weight, the more critical the situation described by the term. For example, in a medical consultation scenario, "acute myocardial infarction" is more critical than "chronic gastritis," so the term weight of "acute myocardial infarction" can be higher than that of "chronic gastritis." As another example, in an industrial manufacturing scenario, the urgency of "equipment failure" is higher than that of "material shortage," so the term weight of "equipment failure" is higher than that of "material shortage."
[0032] For a preset terminology database, the sum of the term weights corresponding to all terms in the database can be calculated and divided by the total number of terms in the database to obtain the average term weight. Based on the average term weight and the term matching degree, the semantic importance parameters of the sample speech data can be calculated. Optionally, a first weight and a second weight can be predefined. The product of the first weight and the term matching is calculated to obtain the first product; the product of the second weight and the average term weight is calculated to obtain the second product, where the second weight is greater than the first weight; the sum of the first product and the second product is used to obtain the semantic importance parameters.
[0033] The first weight and the second weight can be set according to actual needs. In this embodiment of the invention, the first weight can be set to 0.4 and the second weight can be set to 0.6.
[0034] The speech samples in the pre-set speech database are collected by the participants over a historical period. Understandably, different speech samples will have different acoustic features, such as voiceprints, accents, and intonations. For example, the voiceprint features of Mandarin speech are similar to those of most speech samples in the pre-set database, while rare dialects and the distinctive voices of children or the elderly will differ significantly. By matching sample speech data with each speech sample in the pre-set database, the acoustic uniqueness parameters of the sample speech data can be calculated. The closer the acoustic uniqueness parameter is to 0, the stronger the acoustic uniqueness; the closer the acoustic uniqueness parameter is to 1, the weaker the acoustic uniqueness.
[0035] Optionally, when calculating the acoustic unique parameters of the sample speech data, the following steps can be taken: generating speech features corresponding to each speech in a preset speech library; calculating the cosine distance between the acoustic features and each speech feature, and determining the maximum and minimum cosine distances; obtaining the average cosine distance by dividing the sum of all cosine distances by the number of speech features; and calculating the acoustic unique parameters of the sample speech data using the maximum, minimum, and average cosine distances.
[0036] The speech samples in the pre-set speech library are collected locally by the participants. For each speech sample in the pre-set speech library, the corresponding voiceprint features can be extracted using a feature extraction model and recorded as speech features for subsequent use. The cosine distance between the acoustic features and each speech feature is calculated, resulting in multiple cosine distances. Among these multiple cosine distances, the maximum and minimum cosine distances can be determined, and the average cosine distance can also be calculated.
[0037] Finally, the first cosine distance can be obtained by subtracting the minimum cosine distance from the average cosine distance; the second cosine distance can be obtained by subtracting the minimum cosine distance from the maximum cosine distance; the intermediate ratio value can be obtained by dividing the first cosine distance by the second cosine distance; and the acoustic unique parameter can be obtained by subtracting the intermediate ratio value from the specified value. For example, if the average cosine distance is 0.8, the minimum cosine distance is 0.2, and the maximum cosine distance is 0.9, then the acoustic unique parameter is 1 - (0.8 - 0.2) / (0.9 - 0.2) = 0.14.
[0038] Having performed the aforementioned calculations, the semantically important parameters and acoustically unique parameters of the sample speech data have been calculated. These parameters can be used to calculate the noise amplitude matching the sample speech data. Specifically, a reference amplitude can be preset, the size of which can be set according to actual needs. In this embodiment of the invention, the reference amplitude can be set to 0.5.
[0039] The noise amplitude can be calculated by dividing the baseline amplitude by the sum of the semantically important parameter and the acoustically unique parameter. Optionally, to avoid excessive noise injection due to excessive noise, the sum of the semantically important parameter and the acoustically unique parameter can be calculated first as a candidate parameter; if the candidate parameter is less than a specified parameter, the noise amplitude can be directly determined as the preset amplitude. Both the specified parameter and the preset amplitude can be set according to actual needs. In this embodiment of the invention, the specified parameter can be 0.2, and the preset amplitude is 2.5.
[0040] Using the above method, for different sample speech data, the noise amplitude that is suitable for the current sample speech data can be calculated. This will not affect the training effect of the model due to injecting too much noise, nor will it lead to privacy leakage due to injecting too little noise.
[0041] S130. Add noise to the speech representation according to the noise amplitude to obtain a privacy speech representation.
[0042] Based on the calculated noise amplitude, the participants calculate the corresponding noise and then add noise to the speech representation to protect the privacy of the speech representation, thus obtaining a private speech representation.
[0043] One implementation method may involve generating a candidate noise matrix that matches the noise amplitude; calculating an estimated signal-to-noise ratio (SNR) for noise addition processing based on the candidate noise matrix and the speech representation; if the estimated SNR is less than a specified SNR, reducing the noise amplitude by a specified ratio until the estimated SNR is not less than the specified SNR, thus obtaining a target noise matrix; if the estimated SNR is not less than the specified SNR, determining the candidate noise matrix as the target noise matrix; and superimposing the target noise matrix and the speech representation to obtain a privacy-preserving speech representation.
[0044] When adding noise to a speech representation, a candidate noise matrix matching the noise amplitude can be generated. This candidate noise matrix can be a Gaussian noise matrix. To avoid excessive noise injection that could distort the speech representation, the signal-to-noise ratio (SNR) needs to be greater than a specified SNR. Therefore, the estimated SNR for the noise-added processing can be calculated based on the candidate noise matrix and the speech representation.
[0045] The estimated signal-to-noise ratio can be calculated using the following formula: ; Wherein, SNR is the estimated signal-to-noise ratio; z(j) is the speech representation; N is the candidate noise matrix. The larger the signal-to-noise ratio, the smaller the injected noise and the less interference; the smaller the signal-to-noise ratio, the larger the injected noise and the greater the interference.
[0046] If the calculated estimated signal-to-noise ratio (SNR) is less than the specified SNR, it indicates that the injected noise is large, which may lead to distortion of the speech representation. The larger the injected noise, the larger its corresponding noise amplitude. To reduce the injected noise, the noise amplitude needs to be reduced. Optionally, the noise amplitude can be gradually reduced according to a specified ratio until the estimated SNR is not less than the specified SNR, at which point the target noise matrix can be obtained.
[0047] Specifically, the specified ratio can be set according to actual needs. In this embodiment of the invention, the specified ratio is 10%, that is, when reducing the noise amplitude, it can be reduced by 10% and then the estimated signal-to-noise ratio is calculated. If the estimated signal-to-noise ratio is still less than the specified signal-to-noise ratio, it is reduced by another 10% on the basis of the reduction already done once, and the estimated signal-to-noise ratio is calculated again until the calculated estimated signal-to-noise ratio is not less than the specified signal-to-noise ratio. Then the corresponding noise amplitude can be obtained as the target noise amplitude to generate the corresponding target noise matrix.
[0048] If the estimated signal-to-noise ratio calculated in the first calculation is not less than the specified signal-to-noise ratio, it indicates that the injected noise is appropriate, and the aforementioned candidate noise matrix can be directly determined as the target noise matrix.
[0049] By superimposing the target noise matrix and the speech representation, and adding the corresponding noise value to each dimension of the speech representation, a speech representation carrying noise can be obtained, which is the privacy speech representation.
[0050] Understandably, in federated learning, participants and service providers need to transfer data to each other to jointly train the model. In this embodiment of the invention, multiple participants and service providers jointly train a preset model into a speech recognition model. The preset model includes a bottom model and a top model. The bottom model is deployed to the participants, and the top model is deployed to the service provider. After generating privacy-preserving speech representations, the participants can send these privacy-preserving semantic representations to the service provider. The service provider then calculates the loss function and backpropagation gradient of the top model based on the labeled data and the privacy-preserving speech representations sent by all participants, and sends the backpropagation gradient to all participants and adjusts the model parameters of the top model.
[0051] For example, see Figure 3 This diagram illustrates the data interaction between the participants and the service provider. Specifically, after generating private speech representations, multiple participants can send these representations to the service provider. Upon receiving these private speech representations, the service provider can filter them, removing invalid representations. Specifically, private speech representations with a signal-to-noise ratio (SNR) lower than a specified SNR can be removed. Since the participants have already performed filtering using the specified SNR, the service provider performs filtering again; this double filtering ensures that the obtained private speech representations are valid.
[0052] For each participant, a weight can be calculated. Specifically, the service provider can obtain the total number of received privacy speech representations. For a given participant, the weight is calculated by dividing the number of privacy speech representations sent by that participant by the total number. The privacy speech representations of each participant are then fused using a weighted average to obtain a fused privacy representation. This fused privacy representation is then subjected to layer normalization to ensure a consistent distribution of representations across all participants. The representation obtained after normalization can be recorded as the representation to be transmitted.
[0053] The top model may include a feedforward network layer and an output layer. The representation to be transmitted first enters the feedforward network layer for processing, and then enters the output layer to obtain the predicted label. The predicted label can be a label for a specific scenario. For example, in a medical scenario, it may include the name of the disease or the name of the drug, while in an industrial manufacturing scenario, it may include multiple different instruction types.
[0054] The service provider possesses the correct labeled data. It can use the predicted labels and the labeled data to calculate the cross-entropy loss, and then perform backpropagation using an automatic differentiation framework based on the cross-entropy loss. First, it calculates the gradient of the features corresponding to the output layer, and then uses the chain rule to calculate the gradient of the features corresponding to the feedforward network layer, obtaining the final backpropagation gradient. After gradient calculation, L2 norm pruning is performed to avoid gradient explosion that could lead to model training instability.
[0055] The backpropagation gradient is split according to the weights of each participant to obtain the backpropagation gradient for each participant. The service provider then distributes the backpropagation gradient to the corresponding participant so that the participant can use the backpropagation gradient to perform backpropagation and update the model parameters of the bottom model deployed on the participant.
[0056] S140. Calculate the acoustic gradient and semantic gradient based on the backpropagation gradient sent by the service provider.
[0057] When a participant receives the backpropagation gradient from the service provider, it can calculate the acoustic gradient and the semantic gradient. That is, upon receiving the backpropagation gradient, the participant can calculate the gradient of the bottom model parameters based on the chain rule. The gradient corresponding to the attention layer is the acoustic gradient, while the gradient corresponding to the convolutional layer is the semantic gradient.
[0058] S150. Perform privacy protection processing on the sensitive gradients in the acoustic gradient and semantic gradient to obtain the privacy acoustic gradient and privacy semantic gradient.
[0059] Understandably, gradients carry sensitive information about the original data and model structure. To avoid privacy leaks, acoustic and semantic gradients need to be protected. Optionally, privacy protection processing can be performed on sensitive gradients in acoustic and semantic gradients to obtain privacy-preserving acoustic and semantic gradients. This can be achieved by performing sensitivity detection processing on the acoustic and semantic gradients to determine sensitive gradients, including both acoustic and semantic sensitive gradients; pruning the acoustic and semantic sensitive gradients to obtain pruned acoustic and semantic gradients; and injecting specified noise into the pruned acoustic and semantic gradients to obtain privacy-preserving acoustic and semantic gradients, where the specified noise is calculated based on terms in the preset terminology library.
[0060] Sensitive detection processing can identify sensitive gradients from acoustic and semantic gradients. Subsequently, only the sensitive gradients need to be protected for privacy, without processing the entire gradient. This can effectively protect privacy and prevent privacy leaks, while also reducing the amount of data processing.
[0061] Optionally, when performing sensitive detection, the cosine similarity between each feature vector in the semantic gradient and each term in the preset term library can be calculated; feature vectors with a cosine similarity greater than a first preset value can be determined as semantically sensitive gradients; the variance of the voiceprint gradient components in the acoustic gradient can be calculated; and voiceprint gradient components with a variance greater than a second preset value can be determined as acoustically sensitive gradients.
[0062] For each term in the preset terminology library, the term can be converted into a corresponding term vector; the cosine similarity between each feature vector in the semantic gradient and the term vector is calculated. If the cosine similarity between a feature vector and a term vector is greater than a first preset value, it indicates that the feature vector carries specific semantic information and can be identified as a semantically sensitive gradient.
[0063] As previously explained, terms may have corresponding weights. When calculating cosine similarity, these weights can also be used to weight the cosine similarity to obtain the final similarity. If the weighted similarity is greater than a third preset value, the corresponding feature vector can be marked as a semantically sensitive gradient. The third preset value is less than the first preset value. The first and third preset values can be set according to actual needs. In this embodiment of the invention, the first preset value is 0.8, and the third preset value is 0.75.
[0064] Voiceprints are a user's voice fingerprint, a type of sensitive biometric information. Based on acoustic gradients, voiceprint gradient components can be extracted, and the variance of these components can be calculated to measure the fluctuation of the voiceprint gradient. If the variance is greater than a second preset value, the more drastic the fluctuation of the voiceprint gradient, indicating that this part of the voiceprint gradient carries rich voiceprint information. If leaked, it could lead to the disclosure of user identity information, and can be marked as an acoustically sensitive gradient. If the variance is not greater than the second preset value, it indicates that it carries little voiceprint information, is close to general speech characteristics, and will not leak user identity.
[0065] After determining the semantically sensitive gradient and the acoustically sensitive gradient, these gradients can be selectively pruned to obtain pruned acoustic and semantic gradients. As one implementation, pruning the sensitive gradients can be achieved by calculating the Euclidean norms of the semantically and acoustically sensitive gradients to obtain the semantic and acoustic norms. If the semantic norm is greater than a first preset norm, the semantically sensitive gradient is pruned based on the first preset norm to obtain the pruned semantic gradient; if the acoustic norm is greater than a second preset norm, the acoustically sensitive gradient is pruned based on the second preset norm to obtain the pruned acoustic gradient.
[0066] The Euclidean norm, also known as the L2 norm, can be used to calculate the L2 norm of semantically sensitive gradients, the semantic norm, and the acoustic norm. For the semantic norm, a larger semantic norm indicates richer semantic information and a higher risk of leakage. The semantic norm is compared with a first preset norm; if the semantic norm is greater than the first preset norm, the semantically sensitive gradient can be clipped based on the first preset norm. The first preset norm can be set based on practical experience; in this embodiment, it can be set to 0.35.
[0067] Specifically, semantically sensitive gradients can be trimmed according to the following formula: ; in, Characterize the semantically sensitive gradient after clipping; The L2 norm of the clipped semantically sensitive gradient is represented by r1, which represents the first pre-defined norm. For the clipped semantically sensitive gradient, it is necessary to calculate whether its L2 norm is less than or equal to the first pre-defined norm. If the condition is not met, clipping needs to continue until the condition is met, and the final clipped semantically sensitive gradient is obtained.
[0068] Similarly, a larger acoustic norm indicates richer voiceprint information and a higher risk of leakage. The acoustic norm is compared with a second preset norm. If the acoustic norm is greater than the second preset norm, the acoustic sensitive gradient can be trimmed based on the second preset norm. The second preset norm can be set based on practical experience. In this invention, the second preset norm is greater than the first preset norm, and can be set to 0.65.
[0069] Specifically, the acoustic sensitive gradient can be trimmed according to the following formula: ; in, Characterizes the clipped acoustic sensitivity gradient; r1 represents the semantically sensitive gradient; r2 represents the second predefined norm.
[0070] It should be noted that the above only applies to sensitive gradients. While insensitive gradients are not clipped, their L2 norm must be less than 1 to avoid gradient explosion affecting training stability. After obtaining the clipped gradients, specified noise can be injected into the clipped semantic and acoustic gradients.
[0071] The type of noise to be injected can be selected based on the actual application scenario. For example, Laplace noise can be used in medical consultation scenarios with high privacy requirements, while Gaussian noise can be used in industrial manufacturing scenarios with high real-time requirements. Regardless of the type of noise, semantic irrelevance verification is required before injection. That is, the cosine similarity between the noise to be injected and each term in the preset terminology library is calculated. If the cosine similarity is less than a certain similarity, the noise to be injected can be considered semantically irrelevant and can be identified as the specified noise.
[0072] The specific similarity can be set to a small value, such as 0.1. If the similarity between the noise to be injected and the term is greater than this specific similarity, the noise needs to be regenerated until the specified noise is determined. Then, the specified noise is injected into the clipped acoustic gradient and the clipped semantic gradient to obtain the privacy semantic gradient and the privacy acoustic gradient.
[0073] Optionally, to prevent the injected noise from deviating excessively from the original gradient direction and affecting model convergence, the L2 norm of the privacy semantic gradient after injecting the noise needs to be less than a specified multiple of the corresponding pruning threshold. For example, the L2 norm of the privacy semantic gradient needs to be less than a first multiple of a first preset value; the L2 norm of the privacy acoustic gradient needs to be less than a second multiple of a second preset value. If this condition is not met, the injected noise needs to be recalculated until the condition is met, at which point the final privacy semantic gradient and privacy acoustic gradient are obtained. The first and second multiples can be set empirically. S160. Using the privacy acoustic gradient and the privacy semantic gradient, update the model parameters of the preset model to obtain a speech recognition model.
[0074] After obtaining the privacy acoustic gradient and privacy semantic gradient, the parameters of the attention layer can be adjusted based on the privacy acoustic gradient, and the parameters of the convolutional layer can be adjusted based on the privacy semantic gradient. This allows for updating the model parameters of the bottom model deployed on the participants, and when the convergence condition is met, a speech recognition model can be obtained.
[0075] It should be noted that the speech recognition model consists of two parts: a bottom model with adjusted parameters and a top model. After the convergence condition is met, the service provider can send the model parameters of the top model to each participant so that the participants can deploy the complete speech recognition model and use it in real-world application scenarios, such as automatically generating medical records in medical consultations and accurately recognizing user voice commands in industrial manufacturing scenarios.
[0076] Typically, convergence is considered achieved when the loss function stabilizes. Of course, convergence conditions can be specifically set. For example, in this embodiment of the invention, after each training round, the participants calculate the word error rate of the validation set, and calculate the sum of the insertion, deletion, and replacement counts; dividing this sum by the reference word count and multiplying by 100% yields the word error rate. The insertion count refers to the number of irrelevant words the model misidentifies; for example, if the speech is "open device," but the model identifies it as "open new device," then "new" is an inserted word. The deletion count refers to words the model misses; for example, if the speech is "navigate to the company," but the model identifies it as "navigate to," then "company" is missed. The replacement count refers to the number of words the model incorrectly identifies; for example, if the speech is "play music," but the model identifies it as "play song." The reference word count refers to the total number of correct words corresponding to the original speech. A lower word error rate indicates higher speech recognition accuracy.
[0077] If the word error rate calculated after three consecutive rounds of training is less than a preset value, and the decrease in training set loss is less than a preset magnitude, the model can be considered converged. The model parameters of the bottom model and the model parameters of the top model constitute the final speech recognition model. The preset value and preset magnitude can be set according to actual needs. In this embodiment of the invention, the preset value is set to 0.1%, and the preset magnitude is set to 1e-5.
[0078] If this condition is not met, the model can be considered not to have converged. Training can continue, and the corresponding training rounds can be recorded. If the model still has not converged after the specified number of training rounds, the hyperparameters can be adjusted and the model can be retrained.
[0079] The speech recognition model training scheme provided in this invention can be applied to various scenarios. For example, in a medical consultation scenario, multiple hospitals can jointly train a speech recognition model for the medical scenario, while protecting the data privacy of each hospital. Using the trained speech recognition model in the consultation scenario can accelerate the process of smart healthcare. In an industrial manufacturing scenario, multiple industrial manufacturers can jointly train a speech recognition model to ensure the accuracy of speech recognition in this scenario.
[0080] The method provided in this invention can achieve data non-domain protection and precise privacy protection, preventing the leakage of sensitive information at the source in federated learning. Each participant can adaptively inject noise for different speech representations, effectively protecting privacy while avoiding excessive noise injection that reduces data usability and thus affects model training performance. Upon receiving gradients from the service provider, sensitivity detection is performed on the gradients, and sensitive information is differentially pruned, preventing privacy leakage through gradients and balancing the security of speech data and model training performance in federated learning.
[0081] To better implement the above methods, this embodiment of the invention also provides a speech recognition model training device, which can be integrated into an electronic device, such as a terminal or a server. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers.
[0082] For example, in this embodiment, the method of the present invention will be described in detail by taking the example of a speech recognition model training device specifically integrated into a server.
[0083] For example, such as Figure 4 As shown, the speech recognition model device 200 may include a representation extraction module 210, an amplitude calculation module 220, a noise addition module 230, a gradient calculation module 240, a privacy processing module 250, and a parameter update module 260.
[0084] The representation extraction module 210 is used to obtain the speech representation corresponding to the sample speech data using a preset model. The speech representation includes acoustic features and semantic features. The amplitude calculation module 220 is used to calculate the noise amplitude of the sample speech data based on the semantically important parameters and acoustically unique parameters of the sample speech data. The noise-adding module 230 is used to add noise to the speech representation according to the noise amplitude to obtain a privacy speech representation. The gradient calculation module 240 is used to calculate the acoustic gradient and the semantic gradient based on the backpropagation gradient sent by the service provider. The backpropagation gradient is calculated by the service provider based on the label data and the privacy speech representation sent by all participants. Privacy processing module 250 is used to perform privacy protection processing on the sensitive gradients in the acoustic gradient and semantic gradient to obtain privacy acoustic gradient and privacy semantic gradient. The parameter update module 260 is used to update the model parameters of the preset model using the privacy acoustic gradient and the privacy semantic gradient to obtain a speech recognition model.
[0085] In some embodiments, the amplitude calculation module 220 is specifically used for: The sample speech data is matched with each term in a preset terminology database to calculate the semantic importance parameters of the sample speech data; Based on the acoustic features of the sample speech data, the sample speech data is matched with each speech in a preset speech library to calculate the unique acoustic parameters of the sample speech data. The noise amplitude of the sample speech data is calculated based on the semantically important parameters, the acoustically unique parameters, and the reference amplitude.
[0086] In some embodiments, the amplitude calculation module 220 is specifically used for: Convert the sample speech data into sample speech text; The sample speech text is matched with each term in a preset terminology database to obtain the matched terms; The term matching degree is calculated based on the number of matched terms and the total number of words in the sample speech text; Using the term matching degree and the term weight of each term, the semantic importance parameters of the sample speech data are calculated.
[0087] In some embodiments, the amplitude calculation module 220 is specifically used for: Generate speech features for each speech in a preset speech library; Calculate the cosine distance between the acoustic feature and each speech feature, and determine the maximum and minimum cosine distances among them; The average cosine distance is obtained by dividing the sum of all cosine distances by the number of speech features. The acoustic unique parameters of the sample speech data are calculated using the maximum cosine distance, minimum cosine distance, and average cosine distance.
[0088] In some embodiments, the privacy processing module 250 is specifically used for: Sensitive detection processing is performed on the acoustic gradient and semantic gradient to determine the sensitive gradient, which includes the acoustic sensitive gradient and the semantic sensitive gradient. The acoustic sensitive gradient and semantic sensitive gradient are cropped to obtain cropped acoustic gradient and cropped semantic gradient; Predetermined noise is injected into the clipped acoustic gradient and clipped semantic gradient to obtain the privacy acoustic gradient and privacy semantic gradient, wherein the specified noise is calculated based on terms in the preset terminology library.
[0089] In some embodiments, the privacy processing module 250 is specifically used for: Calculate the cosine similarity between each feature vector in the semantic gradient and each term in the predefined term library; The feature vectors whose cosine similarity is greater than a first preset value are determined as semantically sensitive gradients; Calculate the variance of the acoustic gradient component in the acoustic gradient; The acoustic gradient component with a variance greater than the second preset value is determined as the acoustically sensitive gradient.
[0090] In some embodiments, the privacy processing module 250 is specifically used for: Calculate the Euclidean norms of the semantically sensitive gradient and the acoustically sensitive gradient to obtain the semantic norm and the acoustic norm; If the semantic norm is greater than the first preset norm, the semantically sensitive gradient is clipped based on the first preset norm to obtain the clipped semantic gradient; If the acoustic norm is greater than the second preset norm, the acoustic sensitive gradient is clipped based on the second preset norm to obtain the clipped acoustic gradient.
[0091] In some embodiments, the noise-adding module 230 is specifically used for: Generate a candidate noise matrix that matches the noise amplitude; Based on the candidate noise matrix and the speech representation, the estimated signal-to-noise ratio for noise processing is calculated; If the estimated signal-to-noise ratio is less than the specified signal-to-noise ratio, the noise amplitude is reduced by a specified ratio until the estimated signal-to-noise ratio is not less than the specified signal-to-noise ratio, thus obtaining the target noise matrix; If the estimated signal-to-noise ratio is not less than the specified signal-to-noise ratio, the candidate noise matrix is determined as the target noise matrix; The target noise matrix and the speech representation are superimposed to obtain the privacy speech representation.
[0092] In some embodiments, the preset model includes a bottom model deployed to the participants and a top model deployed to the service provider, and the gradient calculation module 240 is specifically used for: The service provider sends the privacy speech representation to the service provider so that the service provider can calculate the loss function and backpropagation gradient of the top model based on the tag data and the privacy speech representation sent by all participants, and send the backpropagation gradient to the participants and adjust the model parameters of the top model. The parameter update module 260 is specifically used for: The model parameters of the bottom model are updated using the privacy acoustic gradient and the privacy semantic gradient until the loss function converges, thus obtaining the speech recognition model.
[0093] In practice, the above modules can be implemented as independent entities or combined in any way to be implemented as the same or several entities. For the specific implementation of the above modules, please refer to the previous method implementation examples, which will not be repeated here.
[0094] As can be seen from the above, the speech recognition model training device in this embodiment can acquire the acoustic and semantic features corresponding to the sample speech data using a preset model. It adaptively calculates the noise amplitude based on the semantically important parameters and unique acoustic parameters of the sample speech data, and adds noise to the speech representation based on the noise amplitude to adaptively protect the privacy of the representation. Based on the backpropagation gradient sent by the service provider, it calculates the acoustic and semantic gradients, identifies sensitive gradients for privacy protection, and then uses the privacy-protected gradients to update the model parameters. By implementing an adaptive noise-adding strategy in both acoustic and semantic dimensions, it can effectively protect acoustic and semantic privacy through noise injection, and precisely control the noise intensity to avoid excessive interference with feature quality. Simultaneously, privacy protection processing is carried out at the gradient level, ultimately achieving a balance between privacy security and the training effect of the speech recognition model.
[0095] This invention also provides an electronic device, which can be a terminal, a server, or other similar devices. The terminal can be a mobile phone, tablet computer, smart Bluetooth device, laptop computer, personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers, etc.
[0096] In some embodiments, the speech recognition model training device can also be integrated into multiple electronic devices, such as multiple servers, with multiple servers implementing the speech recognition model training method of the present invention.
[0097] In this embodiment, a server will be used as an example for detailed description. For example, ... Figure 5 As shown, it illustrates a structural schematic diagram of the electronic device involved in an embodiment of the present invention, specifically: The electronic device may include components such as a processor 310 with one or more processing cores, a memory 320 with one or more computer-readable storage media, a power supply 330, an input module 340, and a communication module 350. Those skilled in the art will understand that... Figure 5 The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 310 is the control center of the electronic device, connecting various parts of the device via various interfaces and lines. It executes various functions and processes data by running or executing software programs and / or modules stored in the memory 320, and by calling data stored in the memory 320. In some embodiments, the processor 310 may include one or more processing cores; in some embodiments, the processor 310 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 310.
[0098] The memory 320 can be used to store software programs and modules. The processor 310 executes various functional applications and data processing by running the software programs and modules stored in the memory 320. The memory 320 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 320 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 320 may also include a memory controller to provide the processor 310 with access to the memory 320.
[0099] The electronic device also includes a power supply 330 that supplies power to the various components. In some embodiments, the power supply 330 can be logically connected to the processor 310 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 330 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.
[0100] The electronic device may also include an input module 340, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.
[0101] The electronic device may also include a communication module 350. In some embodiments, the communication module 350 may include a wireless module, through which the electronic device can perform short-range wireless transmission, thereby providing users with wireless broadband internet access. For example, the communication module 350 can be used to help users send and receive emails, browse web pages, and access streaming media.
[0102] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 310 in the electronic device loads the executable files corresponding to the processes of one or more applications into the memory 320 according to the following instructions, and the processor 310 runs the applications stored in the memory 320, thereby implementing the steps in the methods of the various embodiments of the present invention.
[0103] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.
[0104] As can be seen from the above, the electronic device provided in this embodiment of the invention can use a preset model to obtain the acoustic and semantic features corresponding to the sample speech data. It adaptively calculates the noise amplitude based on the semantically important parameters and unique acoustic parameters of the sample speech data, and adds noise to the speech representation based on the noise amplitude to adaptively protect the privacy of the representation. Based on the backpropagation gradient sent by the service provider, it calculates the acoustic and semantic gradients, identifies sensitive gradients for privacy protection, and then uses the privacy-protected gradients to update the model parameters. By implementing an adaptive noise-adding strategy in both acoustic and semantic dimensions, it can effectively protect acoustic and semantic privacy through noise injection, and precisely control the noise intensity to avoid excessive interference with feature quality. Simultaneously, privacy protection processing is carried out at the gradient level, ultimately achieving a balance between privacy security and the training effect of the speech recognition model.
[0105] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.
[0106] To this end, embodiments of the present invention provide a computer-readable storage medium storing a plurality of instructions which can be loaded by a processor to execute steps in any of the speech recognition model training methods provided in embodiments of the present invention.
[0107] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.
[0108] According to one aspect of the present invention, a computer program product or computer program is provided, comprising a computer program / instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer program / instructions from the computer-readable storage medium and executes the computer program / instructions, causing the electronic device to perform the methods provided in various optional implementations of the privacy protection or model training aspects provided in the above embodiments.
[0109] Since the instructions stored in the storage medium can execute the steps in any of the speech recognition model training methods provided in the embodiments of the present invention, the beneficial effects that any of the speech recognition model training methods provided in the embodiments of the present invention can achieve can be realized, as detailed in the preceding embodiments, and will not be repeated here.
[0110] The above provides a detailed description of a speech recognition model training method and electronic device provided by the embodiments of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.
Claims
1. A method for training a speech recognition model, characterized in that, The method includes: A preset model is used to obtain the speech representation corresponding to the sample speech data, and the speech representation includes acoustic features and semantic features. Based on the semantically important parameters and acoustically unique parameters of the sample speech data, the noise amplitude of the sample speech data is calculated. The speech representation is noise-added based on the noise amplitude to obtain a privacy speech representation. Based on the backpropagation gradient sent by the service provider, the acoustic gradient and semantic gradient are calculated. The backpropagation gradient is calculated by the service provider based on the labeled data and the privacy speech representation sent by all participants. Privacy protection processing is performed on the sensitive gradients in the acoustic gradient and semantic gradient to obtain the privacy acoustic gradient and privacy semantic gradient. The model parameters of the preset model are updated using the privacy acoustic gradient and the privacy semantic gradient to obtain a speech recognition model.
2. The method according to claim 1, characterized in that, The calculation of the noise amplitude of the sample speech data based on the semantically important parameters and acoustically unique parameters of the sample speech data includes: The sample speech data is matched with each term in a preset terminology database to calculate the semantic importance parameters of the sample speech data; Based on the acoustic features of the sample speech data, the sample speech data is matched with each speech in a preset speech library to calculate the unique acoustic parameters of the sample speech data. The noise amplitude of the sample speech data is calculated based on the semantically important parameters, the acoustically unique parameters, and the reference amplitude.
3. The method according to claim 2, characterized in that, The process of matching the sample speech data with each term in a preset terminology database to calculate the semantic importance parameters of the sample speech data includes: Convert the sample speech data into sample speech text; The sample speech text is matched with each term in a preset terminology database to obtain the matched terms; The term matching degree is calculated based on the number of matched terms and the total number of words in the sample speech text; Using the term matching degree and the term weight of each term, the semantic importance parameters of the sample speech data are calculated.
4. The method according to claim 2, characterized in that, The step of matching the sample speech data with each speech in a preset speech library based on the acoustic features of the sample speech data to calculate the unique acoustic parameters of the sample speech data includes: Generate speech features for each speech in a preset speech library; Calculate the cosine distance between the acoustic feature and each speech feature, and determine the maximum and minimum cosine distances among them; The average cosine distance is obtained by dividing the sum of all cosine distances by the number of speech features. The acoustic unique parameters of the sample speech data are calculated using the maximum cosine distance, minimum cosine distance, and average cosine distance.
5. The method according to claim 1, characterized in that, The privacy-preserving processing of the acoustic gradient and the sensitive gradient in the semantic gradient to obtain the privacy acoustic gradient and the privacy semantic gradient includes: Sensitive detection processing is performed on the acoustic gradient and semantic gradient to determine the sensitive gradient, which includes the acoustic sensitive gradient and the semantic sensitive gradient. The acoustic sensitive gradient and semantic sensitive gradient are cropped to obtain cropped acoustic gradient and cropped semantic gradient; Injecting specified noise into the clipped acoustic gradient and clipped semantic gradient yields privacy acoustic gradient and privacy semantic gradient, wherein the specified noise is calculated based on terms in the preset terminology library.
6. The method according to claim 5, characterized in that, The sensitive detection processing of the acoustic gradient and semantic gradient to determine the sensitive gradient includes: Calculate the cosine similarity between each feature vector in the semantic gradient and each term in the predefined term library; The feature vectors whose cosine similarity is greater than a first preset value are determined as semantically sensitive gradients; Calculate the variance of the acoustic gradient component in the acoustic gradient; The acoustic gradient component with a variance greater than the second preset value is determined as the acoustically sensitive gradient.
7. The method according to claim 5, characterized in that, The process of cropping the acoustically sensitive gradient and the semantically sensitive gradient to obtain cropped acoustic gradient and cropped semantic gradient includes: Calculate the Euclidean norms of the semantically sensitive gradient and the acoustically sensitive gradient to obtain the semantic norm and the acoustic norm; If the semantic norm is greater than the first preset norm, the semantically sensitive gradient is clipped based on the first preset norm to obtain the clipped semantic gradient; If the acoustic norm is greater than the second preset norm, the acoustic sensitive gradient is clipped based on the second preset norm to obtain the clipped acoustic gradient.
8. The method according to claim 1, characterized in that, The step of adding noise to the speech representation based on the noise amplitude to obtain a privacy speech representation includes: Generate a candidate noise matrix that matches the noise amplitude; Based on the candidate noise matrix and the speech representation, the estimated signal-to-noise ratio for noise processing is calculated; If the estimated signal-to-noise ratio is less than the specified signal-to-noise ratio, the noise amplitude is reduced by a specified ratio until the estimated signal-to-noise ratio is not less than the specified signal-to-noise ratio, thus obtaining the target noise matrix; If the estimated signal-to-noise ratio is not less than the specified signal-to-noise ratio, the candidate noise matrix is determined as the target noise matrix; The target noise matrix and the speech representation are superimposed to obtain the privacy speech representation.
9. The method according to claim 1, characterized in that, The preset model includes a bottom model deployed to the participants and a top model deployed to the service provider. After adding noise to the speech representation based on the noise amplitude to obtain a privacy speech representation, the method further includes: The service provider sends the privacy speech representation to the service provider so that the service provider can calculate the loss function and backpropagation gradient of the top model based on the tag data and the privacy speech representation sent by all participants, and send the backpropagation gradient to all participants and adjust the model parameters of the top model. The step of updating the model parameters of the preset model using the privacy acoustic gradient and the privacy semantic gradient to obtain the speech recognition model includes: The model parameters of the bottom model are updated using the privacy acoustic gradient and the privacy semantic gradient until the loss function converges, thus obtaining the speech recognition model.
10. An electronic device, characterized in that, The method includes a processor and a memory, the memory storing multiple instructions; the processor loads instructions from the memory to perform the steps in the speech recognition model training method as described in any one of claims 1 to 9.