Audio processing method and electronic equipment

Abstract

The invention discloses an audio processing method and electronic equipment, and belongs to the field of artificial intelligence. The method is executed by the electronic equipment, the electronic equipment is deployed with an end-side counterfeiting detection model, and the method comprises the following steps: acquiring a first audio to be processed; performing forgery detection on the first audio through the end-side forgery detection model to obtain a detection result, the detection result being used for indicating whether the first audio is a forgery audio; wherein the end side counterfeit detection model is a lightweight model obtained by performing compression processing on a large cloud counterfeit detection model.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence, specifically relating to an audio processing method and an electronic device. Background Technology

[0002] In recent years, deep learning-based speech synthesis and voice cloning technologies have developed rapidly. They can quickly replicate the voice of a target speaker using a small number of speech samples and generate high-fidelity, natural-sounding personalized voices. They have broad application prospects in fields such as virtual humans, intelligent customer service, education, and entertainment.

[0003] However, this technology also brings serious security risks. Synthesized speech is easily used for illegal activities such as fraud and identity impersonation, threatening personal privacy and public safety. Existing protection methods mostly rely on high-performance computing resources in the cloud, such as cloud-based detection of forged speech, large-scale adversarial training, and complex watermark embedding. However, cloud-based detection and processing usually require network transmission and remote computing, introducing additional latency, making it difficult to meet the extremely low-latency requirements of applications such as voice interaction and intelligent assistants. Moreover, cloud solutions require uploading audio to remote servers, increasing the risk of audio data being stolen or misused during transmission and storage. Summary of the Invention

[0004] The purpose of this application is to provide an audio processing method, electronic device, storage medium, and program product that can detect audio forgery in electronic devices, thereby solving the problems of poor real-time performance and high risk associated with cloud-based audio processing.

[0005] In a first aspect, embodiments of this application provide an audio processing method executed by an electronic device, wherein the electronic device is equipped with an end-side forgery detection model, and the method includes: Get the first audio file to be processed; The first audio is subjected to forgery detection using the edge-side forgery detection model to obtain a detection result, which is used to indicate whether the first audio is forged audio; wherein, the edge-side forgery detection model is a lightweight model obtained by compressing the large cloud-based forgery detection model.

[0006] Secondly, embodiments of this application provide an electronic device, which is equipped with an end-side forgery detection model, and the electronic device further includes: The acquisition module is used to acquire the first audio file to be processed. The forgery detection module is used to perform forgery detection on the first audio using the terminal forgery detection model and obtain a detection result. The detection result is used to indicate whether the first audio is forged audio. The terminal forgery detection model is a lightweight model obtained by compressing the large cloud-based forgery detection model.

[0007] Thirdly, embodiments of this application provide an electronic device including a processor and a memory, wherein the memory stores programs or instructions executable on the processor, and the programs or instructions, when executed by the processor, implement the steps of the method described in the first aspect.

[0008] Fourthly, embodiments of this application provide a readable storage medium on which a program or instructions are stored, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0009] Fifthly, embodiments of this application provide a chip, the chip including a processor and a communication interface, the communication interface being coupled to the processor, the processor being used to run programs or instructions to implement the method as described in the first aspect.

[0010] In a sixth aspect, embodiments of this application provide a computer program product stored in a storage medium, which is executed by at least one processor to implement the method described in the first aspect.

[0011] In this embodiment, by performing forgery detection on the first audio file to be processed, attackers can be prevented from using synthesized speaker voices to deceive and perform voice imitation or other processing. Furthermore, by using an edge-side forgery detection model deployed locally on the electronic device to perform forgery detection on the first audio file, there is no need to upload the first audio file to the cloud. This avoids the latency caused by uploading audio to the cloud, improves the real-time performance of audio processing, and also avoids the risks of theft or misuse that would result from uploading audio to the cloud, thus enhancing the security of audio processing. Attached Figure Description

[0012] Figure 1 This is a flowchart illustrating an audio processing method provided in some embodiments of this application; Figure 2 This is a flowchart illustrating an audio processing method provided in some embodiments of this application; Figure 3 This is a flowchart illustrating an audio processing method provided in some embodiments of this application; Figure 4 This is a schematic diagram of redundant coding provided in some embodiments of this application; Figure 5 These are schematic diagrams of electronic devices provided in some embodiments of this application; Figure 6 These are schematic diagrams of electronic devices provided in some embodiments of this application; Figure 7 These are schematic diagrams of the structure of electronic devices provided in some embodiments of this application. Detailed Implementation

[0013] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0014] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0015] Before providing a further detailed description of the embodiments of the present invention, the terms and concepts involved in the embodiments of the present invention will be explained. The terminology used in the implementation section of this application is only used to explain the specific embodiments of this application and is not intended to limit this application. The terminology involved in the embodiments of this application will be explained below.

[0016] Bit stream: A binary sequence consisting of 0s and 1s, usually transmitted in the form of electrical signals in computer networks.

[0017] BCH (Bose-Chaudhuri-Hocquenghem Code): A cyclic error-correcting code based on algebraic construction that can precisely correct a fixed number of bit errors.

[0018] LDPC (Low Density Parity Check Code): A linear block code based on a sparse parity check matrix, which uses iterative message passing to correct errors close to the Shannon limit.

[0019] MSE_LOSS (Mean Squared Error Loss): The mean squared error loss is used to calculate the average of the squared errors between the model's predicted values ​​and the true values.

[0020] CE_LOSS (Cross-Entropy Loss): Cross-entropy loss, used to calculate the difference between the model's predicted distribution and the true distribution.

[0021] KL_LOSS (Kullback-Leibler): Divergence, also known as relative entropy, is used to measure the difference between two probability distributions.

[0022] The audio processing method, apparatus, electronic device, storage medium, and program product provided in this application will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0023] The audio processing method provided in this application can be applied to sound replication scenarios. The following section combines... Figures 1-4 The audio processing method provided in the embodiments of this application will be described in detail. It should be noted that the audio processing method provided in the embodiments of this application is executed by an electronic device, which may include, but is not limited to, user terminal devices such as mobile phones, tablets, desktop computers, and laptops. This application embodiment uses the execution of the audio processing method by an electronic device as an example to illustrate the audio processing method provided in the embodiments of this application.

[0024] See Figure 1 Some embodiments of this application provide audio processing methods, which are executed by an electronic device equipped with an end-side forgery detection model, such as... Figure 1 As shown, the audio processing method includes the following steps 110-120, which will be explained in detail below.

[0025] Step 110. Obtain the first audio file to be processed.

[0026] In some embodiments of this application, the first audio may be audio collected by an electronic device through a sensor such as a microphone. For example, the audio of a speaker collected through a microphone.

[0027] In some embodiments of this application, the first audio file can be audio stored in an audio storage application of an electronic device. Exemplarily, the audio storage application may include, but is not limited to, a recording app, a file management app, and an audio playback app. Users can select audio from the audio storage application as the first audio file to be processed based on their actual needs.

[0028] Step 120. Perform forgery detection on the first audio using the end-side forgery detection model to obtain the detection result, which is used to indicate whether the first audio is forged audio.

[0029] Forged audio refers to synthesized audio created using technical means to imitate or impersonate the voice of a real person. For example, audio generated through voice replication technology.

[0030] In some embodiments of this application, the edge-side spoofing detection model is a lightweight model obtained by compressing the large cloud-based spoofing detection model. This ensures that the spoofing detection model can run in real-time on electronic devices with limited computing power, with low computational overhead. The large cloud-based spoofing detection model is a model deployed in the cloud for audio spoofing detection.

[0031] In some embodiments of this application, the cloud-based forgery detection model can be a pre-trained model for detecting audio forgeries based on a training set. The training data in the training set includes audio samples and their ground truth labels, where the ground truth labels indicate whether an audio sample is forged.

[0032] After acquiring the first audio, the electronic device inputs the first audio into the edge-side spoofing detection model deployed locally on the electronic device. The edge-side spoofing detection model performs spoofing detection on the first audio to determine whether the first audio is spoofed, thereby obtaining the detection result of the first audio.

[0033] In some embodiments of this application, if the detection result of the first audio indicates that the first audio is fake audio, further processing such as sound replication of the first audio can be stopped to prevent attackers from using synthesized speaker voices to deceive and perform voice imitation and other processing.

[0034] In this embodiment, by performing forgery detection on the first audio file to be processed, attackers can be prevented from using synthesized speaker voices to deceive and perform voice imitation or other processing. Furthermore, by using an edge-side forgery detection model deployed locally on the electronic device to perform forgery detection on the first audio file, there is no need to upload the first audio file to the cloud. This avoids the latency caused by uploading audio to the cloud, improves the real-time performance of audio processing, and also avoids the risks of theft or misuse that would result from uploading audio to the cloud, thus enhancing the security of audio processing.

[0035] In some embodiments of this application, the edge-side forgery detection model can be generated by performing model compression processing such as distillation and sparsification on the large cloud-based forgery detection model. Specifically, the edge-side forgery detection model can be generated using the following steps 210-230.

[0036] Step 210. Construct the student model.

[0037] In some embodiments of this application, the student model is a lightweight model with fewer parameters than the large cloud-based forgery detection model. Compared to the large cloud-based forgery detection model, the student model has fewer parameters and a simpler structure. The goal of the student model is to significantly improve inference speed and reduce computational resource consumption while maintaining the highest possible performance.

[0038] In some embodiments of this application, a lightweight model with the same architecture as the cloud-based forgery detection large model can be constructed as a student model. For example, the structure of the cloud-based forgery detection large model can be represented by the following formula (1):

[0039] In the above formula (1), Representation model, This represents the input data of the model. Indicates the number of hidden layers. Indicates the dimension of the hidden layer. Indicates the model output, Indicates the output layer. Indicates a hidden layer. , Represents the activation function, for example, It can be a linear rectification function (ReLU). This represents the bias term of the output layer. This represents the bias term of the hidden layer. The dimension of the hidden layer in the student model can be smaller than that of the large cloud-based forgery detection model. Assume that the large cloud-based forgery detection model... Set as Then the student model Can be set to .

[0040] Step 220. Use the cloud-based forgery detection model as the teacher model for the student model to train the student model and obtain the initial forgery detection model.

[0041] After obtaining the student model, the cloud-based forgery detection model can be used as the teacher model for the student model. The student model can be trained through knowledge distillation to obtain the initial forgery detection model.

[0042] In some embodiments of this application, step 220 may include steps 2201-2204.

[0043] Step 2201. Input the training data into the cloud-based forgery detection big model and the student model respectively to obtain the first intermediate feature output by each hidden layer in the cloud-based forgery detection big model, the first final output result of the cloud-based forgery detection big model, the second intermediate feature output by each hidden layer in the student model, and the second final output result of the student model.

[0044] The training data includes audio samples and their real labels, which indicate whether the audio samples are fake.

[0045] Step 2202. Calculate the loss value of the student model based on the loss function, the first intermediate feature, the first final output result, the second intermediate feature, and the second final output result.

[0046] In some embodiments of this application, a loss function is predefined before training the student model. The training objective of the student model can be guided by three parts of loss: hidden feature loss, label loss, and distillation loss.

[0047] Hidden feature loss measures the difference between the features output by each layer of the student model and the features output by each layer of the large model. Hidden feature loss allows the student model to mimic the representations or features of the hidden layers (intermediate layers) within the large model used for forgery detection in the cloud. Hidden feature loss can utilize methods such as mean squared error loss and cosine similarity loss.

[0048] For example, taking the hidden feature loss using the mean squared error loss as an example, the hidden feature loss is as shown in equation (3):

[0049] in, This represents the hidden feature loss. This represents the mean squared error loss function. Indicates the feature dimension transformation network The transformed cloud-based forgery detection model The intermediate features output by each hidden layer The student model represents the first The intermediate features output by each hidden layer.

[0050] Label loss measures the difference between the probability distribution of the student model's output and the true labels in the training data, i.e., the "hard labels." Label loss ensures that the student model doesn't completely deviate from the correct answer. Label loss can use cross-entropy loss, mean squared error loss, cosine similarity loss, etc.

[0051] For example, taking the label loss as an example using cross-entropy loss, the label loss is as shown in equation (4):

[0052] In the above formula (4), Indicates label loss, Represents the cross-entropy loss function. This represents the final output of the student model. Taking a student module with 12 hidden layers as an example, ,in, This represents the feature output by the last hidden layer of the student model. This represents the output layer of the student model. This represents the bias term of the last hidden layer in the student model. Represents the true labels of the training data. , where (1,0) indicates that the audio sample input to the student model is real audio, and (0,1) indicates that the audio sample input to the student model is fake audio.

[0053] Distillation loss measures the difference between the probability distribution of the student model's output and the output of a large cloud-based forgery detection model, i.e., the "soft labels." Distillation loss forces the student model to learn the judgment style and inter-category relationships of the teacher model. Distillation loss can be determined using metrics such as KL divergence.

[0054] For example, taking the distillation loss using KL divergence as an example, the distillation loss is shown in equation (5) below:

[0055] In the above formula (5), Indicates distillation loss, Denotes the KL divergence function. This represents the probability distribution output by the cloud-based forgery detection model. This represents the probability distribution of the student model's output.

[0056] The three losses mentioned above are combined with certain weights into a total loss function, which is used to train the student model. Thus, the loss function is a comprehensive loss function including hidden feature loss, label loss, and distillation loss. Specifically, the hidden feature loss is used to determine the difference between the second intermediate feature output of each hidden layer of the student model and the first intermediate feature output of each hidden layer of the cloud-based forgery detection model; the label loss is used to determine the difference between the second final output of the student model and the true label in the training data; and the distillation loss is used to determine the difference between the second final output of the student model and the first final output of the student model.

[0057] For example, the loss function is shown in the following formula (6):

[0058] In the above formula (6), Represents the loss function. This indicates the weight of the distillation loss term. The weights represent the hidden feature loss term. This represents the temperature parameter, used to control the smoothness of the output probability distribution.

[0059] In some embodiments of this application, The value can gradually decrease as the number of training steps increases. For example, as the number of training steps increases, The value can be gradually decreased from 0.7 to 0.1.

[0060] In some embodiments of this application, The value can gradually decrease as the number of training steps increases. For example, as the number of training steps increases, The value can be gradually decreased from 0.4 to 0.1.

[0061] In some embodiments of this application, The value can gradually decrease as the number of training steps increases. For example, as the number of training steps increases, The value can be gradually decreased from 3 to 1.

[0062] This dynamic adjustment during student model training , and / or The adaptive teaching strategy makes the knowledge distillation process more intelligent and efficient, avoids getting stuck in local optima, and maintains the stability and convergence of training.

[0063] In some embodiments of this application, the hidden layer outputs of the cloud-based forgery detection model and the student model have different dimensions. To calculate the hidden feature loss, it is necessary to ensure that the hidden layers of the cloud-based forgery detection model and the student model are dimensionally consistent so that they can be directly compared or matched. Therefore, when training the student model, a feature transformation layer is used to map the dimensions of the student model to be consistent with the dimensions of the cloud-based forgery detection model. The feature transformation layer can be represented by the formula (7) shown in the figure:

[0064] In the above formula (7), Represents the feature transformation layer. This represents a feature dimension transformation network. This represents the input dimension of the feature dimension transformation network. This represents the output dimension of the feature dimension transformation network. When performing dimension transformation on the student model... This represents the dimension of the student model. This indicates the dimensions of the large-scale cloud-based forgery detection model. For example, This means that the dimensions of the student model will be changed from 128 to 512, which is consistent with the dimensions of the cloud-based forgery detection model.

[0065] After obtaining the first intermediate feature, the first final output result, the second intermediate feature, and the second final output result, the corresponding data can be substituted into the loss function to obtain the loss value of the student model.

[0066] Step 2203. If the loss value of the student model is greater than the loss threshold, adjust the parameters of the student model and execute the steps of inputting the training data into the cloud-based forgery detection model and the student model respectively to obtain the first intermediate feature of each hidden layer in the cloud-based forgery detection model, the first final output result of the cloud-based forgery detection model, the second intermediate feature of each hidden layer in the student model, and the second final output result of the student model, until the loss value of the student model is less than the loss threshold.

[0067] In some embodiments of this application, the loss threshold can be set according to actual needs, and no specific limitation is made thereto.

[0068] Step 2204. If the loss value of the student model is less than the loss threshold, the trained student model is determined as the initial forgery detection model.

[0069] Thus, through the above training, a lightweight initial forgery detection model with a parameter count reduced by several times compared to the large cloud-based forgery detection model can be obtained.

[0070] Step 230. Perform parameter sparsification on the initial forgery detection model to obtain the end-side forgery detection model.

[0071] In some embodiments of this application, to further reduce the number of model parameters, after obtaining the initial forgery detection model, the parameters of the initial forgery detection model are further sparsified to obtain the edge forgery detection model. The model parameter sparsification process refers to using a series of techniques to reduce most of the parameters in the model to zero, thereby reducing the model's storage and computational overhead while maintaining model performance as much as possible.

[0072] In some embodiments of this application, in step 230 above, an edge-side forgery detection model can be obtained by sparsely training the initial forgery detection model and sparsely processing the weight matrix of the initial forgery detection model during training. The sparse processing of the weight matrix of the initial forgery detection model may include the following steps 2301-2304.

[0073] Step 2301. Convert the weight matrix into a sparse weight matrix. In the sparse weight matrix, the values ​​of all weight parameters are 0, except for the weight parameters at the diagonal positions, which remain consistent with the weight matrix.

[0074] In some embodiments of this application, when the weight matrix of the initial forgery detection model is a square matrix with the same number of rows and columns, all weight parameters except for the weight parameters at the diagonal positions can be set to 0 to obtain a sparse weight matrix.

[0075] For example, the weight matrix is ​​a 3*3 square matrix as shown in equation (8):

[0076] The sparse weight matrix obtained based on the above weight matrix is ​​shown in equation (9) below:

[0077] In some embodiments of this application, when the weight matrix of the initial forgery detection model is not a square matrix, the weight matrix can be divided into multiple sub-square matrices. Then, for each sub-square matrix, all weight parameters except the weight parameters at the diagonal positions are set to 0 to obtain the sparse sub-square matrix corresponding to each sub-square matrix. The matrix composed of multiple sparse sub-square matrices is determined as the sparse weight matrix.

[0078] For example, the weight matrix is ​​a 3*6 matrix as shown in equation (10):

[0079] The weight matrix can be first divided into two 3*3 sub-matrices as shown in equations (11) and (12):

[0080] Based on the two sub-matrixes mentioned above, we can obtain two 3*3 sparse sub-matrixes as shown in equations (13) and (14):

[0081] Based on the two sparse submatrices mentioned above, the sparse weight matrix shown in equation (15) can be obtained:

[0082] Step 2302. Divide the sparse weight matrix into at least two weight parameter data blocks by row.

[0083] In some embodiments of this application, the sparse weight matrix can be divided into at least two weight parameter data blocks of a preset size by rows. The preset size can be determined based on the processing power of the computing unit of the electronic device. For example, the sparse weight matrix can be divided into at least two weight parameter data blocks of size 16 by rows, with each weight parameter data block containing 16 weight parameters.

[0084] Step 2303. Calculate the sum of squares of all weight parameters in each weight parameter data block.

[0085] In some embodiments of this application, the sum of squares of all weight parameters in each weight parameter data block is calculated. ,in, Indicates the index of the weight parameter data block.

[0086] Step 2304. Set the parameters of the weight parameter data blocks located in the first P positions of the data block sequence to 0. The data block sequence is a sequence obtained by sorting at least two weight parameter data blocks in ascending order of their sum of squares, where P is a positive integer.

[0087] In some embodiments of this application, for at least two weight parameter data blocks, according to the sum of squares The data blocks are sorted in ascending order to obtain a sequence. The weight parameters in the first P weight parameter blocks of the sequence are set to 0, while the weight parameters in the other weight parameter blocks remain unchanged. This results in the sparsed weight parameters. Here, P is a positive number.

[0088] In some embodiments of this application, the value of P can be determined based on the sparsity rate and the total number of weight parameter data blocks. The sparsity rate refers to the proportion of weight parameter data blocks that need to be set to zero. For example, if there are 100 weight parameter data blocks, and 90 of them are set to zero, the sparsity rate is 0.9 or 90%. During sparse training, the sparsity rate can be dynamically adjusted according to the number of training steps. For example, in the warm-up phase of sparse training, sparsity processing can be omitted, and training can proceed normally. After training reaches the last step of the warm-up phase (warmup_step), sparsity processing begins in subsequent training steps, and the sparsity process described in steps 2301-2304 is performed according to the specified number of sparsity steps (sparse_steps). As the number of training steps increases, the sparsity rate gradually increases from 0 to the target sparsity rate (target_sparse_rate). The `warmup_step`, `sparse_steps`, and target sparsity rate can be set according to actual needs. For example, if `warmup_step` is 2000, `sparse_steps` is 10000, and `target_sparse_rate` is 0.8, then sparsification can begin after 2000 training steps, and the sparsity rate can be gradually increased from 0 to 0.8 between steps 20001 and 10000. Furthermore, after the sparsity rate reaches `target_sparse_rate`, sparsification can be performed on the model parameters every N training steps, according to the `target_sparse_rate` sparsity rate. Here, N is a positive integer, and its value can be set according to actual needs; for example, N can be set to 10, without specific limitations. Based on the above training method, before each execution of step 2304 during the sparse training process, the following steps can be performed: Determine the number of training steps for sparse training; determine the sparsity ratio corresponding to the number of training steps; and determine the value of P based on the sparsity ratio and the total number of data blocks for weight parameters.

[0089] In some embodiments of this application, the product of the sparsity rate and the total number of weight parameter data blocks can be used to determine the value of P. For example, if the total number of weight parameter data blocks is 100 and the sparsity rate is 0.9, then the value of P is 90.

[0090] During the inference process of the edge-side forgery detection model obtained in the above manner, the diagonal of the model weights can be extracted and multiplied by the input features to obtain the preliminary feature feature1; the non-zero blocks of the model weights can be extracted and multiplied by the input features in a block-by-block vector multiplication to obtain the feature feature2; the final model output result is feature = feature1 + feature2. After this, the computational cost of the model is reduced by target_sparse_rate.

[0091] By combining knowledge distillation and sparsification methods to construct an edge-side spoofing detection model, the number of parameters in the edge-side spoofing detection model can be significantly reduced compared to the number of parameters in the cloud-based spoofing detection model. This greatly reduces the computational overhead when performing audio processing based on the edge-side spoofing detection model, ensuring that the edge-side spoofing detection model can run in real time on electronic devices.

[0092] In some embodiments of this application, the electronic device also deploys an end-side sound replication model; based on this, see [link to relevant documentation]. Figure 2 The audio processing method provided in this application embodiment may further include the following step 130 after the above step 120.

[0093] Step 130. If the detection result indicates that the first audio is not fake audio, generate the second audio based on the first audio and the acquired text using the edge sound replication model.

[0094] If the detection result obtained by the edge-side forgery detection model indicates that the first audio is not forged, the first audio is input into the edge-side sound replication model. Based on the first audio and the acquired text, the edge-side sound replication model generates a second audio with a pronunciation style highly similar to that of the first audio. In this way, the voice of the speaker of the first audio can be replicated.

[0095] In some embodiments of this application, the obtained text may be any text entered by the user or any text downloaded by the user from the internet, and there is no specific limitation thereto.

[0096] In some embodiments of this application, the edge-side sound replication model may include a timbre extraction module, a prosody extraction module, and an audio generation module. The timbre extraction module is used to extract the timbre information of the audio, the prosody extraction module is used to extract the prosody information of the audio, and the audio generation module is used to convert text into audio. Based on this, step 130 above may include the following steps 1301-1303.

[0097] Step 1301. Extract the timbre information of the first audio file using the timbre extraction module.

[0098] In some embodiments of this application, if the first audio is not fake audio, it means that the first audio is the speaker's real audio recording. Based on this, the speaker's timbre information can be extracted from the first audio using a timbre extraction module.

[0099] In some embodiments of this application, the timbre extraction module can extract acoustic features from the first audio to obtain timbre information of the first audio. The timbre information may include extracted acoustic features that represent the speaker's timbre in the first audio, such as Mel-Frequency Cepstral Coefficients (MFCCs) and Linear Predictive Coding (LPC).

[0100] Step 1302. Extract the prosodic information of the first audio using the prosodic extraction module.

[0101] In some embodiments of this application, prosodic information is information that can represent the intonation and rhythm of the speaker's pronunciation of the first audio, thereby determining the speaker's vocal characteristics of the first audio, such as a broadcasting style, a mixed-tone voice, etc.

[0102] In some embodiments of this application, prosodic information may include features that reflect the speaker's rhythm, stress, intonation, etc., of the first audio. For example, prosodic information may include fundamental frequency features, energy features, duration features, and contour features of the first audio. The fundamental frequency features may reflect the speaker's intonation variations and pitch patterns, the energy features may reflect the speaker's stress and emphasis patterns, the duration features may reflect the speaker's speech rate and rhythm patterns, and the contour features may reflect the speaker's overall intonation trend.

[0103] Step 1303. Generate a second audio file based on text, timbre information, and prosody information using the audio generation module.

[0104] In some embodiments of this application, the timbre information of the first audio extracted by the timbre extraction module, the prosody information of the first audio extracted by the prosody extraction module, and the acquired text are input together into the audio generation module. The audio generation module then combines the acquired text with the timbre information and prosody information of the first audio to generate a second audio that corresponds to the text content and has a pronunciation style highly similar to the first audio.

[0105] For example, the first audio is a recording of the speaker, specifically a clear and articulate audio clip in the style of a news anchor, with arbitrary content. If the acquired text is the user's input, "The weather is really nice today; the sunshine is gentle on the streets, the breeze carries the fragrance of flowers, and the whole city seems exceptionally light and bright," then the final generated second audio will be an audio clip with the same content, "The weather is really nice today; the sunshine is gentle on the streets, the breeze carries the fragrance of flowers, and the whole city seems exceptionally light and bright," pronounced in the style of a news anchor.

[0106] In some embodiments of this application, in order to prevent the second audio from being used for illegal activities such as fraud and identity impersonation, reduce the risk of voice data leakage, improve information security, and ensure that the generated audio is traceable, the end-side sound replication model may also include a watermark embedding module. The watermark embedding module is used to write a watermark into the second audio during the generation of the second audio. The watermark is a binary bit stream.

[0107] In some embodiments of this application, the watermark can be a binary bit stream obtained by encoding watermark information such as electronic device identifier ID, timestamp, user authentication status, and geographical location.

[0108] In some embodiments of this application, the end-side sound replication model can synchronously embed a watermark through a watermark embedding module during the generation of the second audio, avoiding additional post-processing steps and ensuring low latency.

[0109] In some embodiments of this application, a frequency band that is not easily perceived by the human ear (such as 100Hz–1500Hz) can be selected as the watermark embedding range in the second audio. The watermark is added only in the watermark embedding range to ensure the imperceptibility of the watermark, improve the security of the watermark, and reduce the impact of the watermark on the listening experience of the second audio.

[0110] In some embodiments of this application, the watermark embedding module can add a watermark by superimposing or subtracting a perturbation on the frequency band energy of the second audio. For example, when the bit value in the watermark binary bitstream is 1, a perturbation can be superimposed on the frequency band energy; when the bit value is 0, a perturbation can be subtracted from the frequency band energy. The magnitude of the perturbation energy can be determined based on the time-domain amplitude of the frequency band position where the watermark is written in the audio, and the magnitude of the perturbation superimposed at different frequency band positions is different. For example, if the time-domain amplitude of the frequency band position is represented as A, the magnitude of the perturbation energy can be set to... Where 'a' is a value greater than 0 and less than 0.01, for example, Therefore, by setting the disturbance energy to a relatively small level, watermark writing can be achieved without affecting audio quality.

[0111] In some embodiments of this application, in order to enable the edge sound replication model to run efficiently in electronic devices, the embedding parameters of the watermark can be dynamically and adaptively adjusted based on the computing power of the electronic device. Specifically, see [link to relevant documentation]. Figure 3 Before step 130 above, step 140 can be performed first.

[0112] Step 140. Determine a watermark embedding strategy that matches the computing power of the electronic device. The watermark embedding strategy includes the watermark bit rate.

[0113] In some embodiments of this application, the watermark is a binary bit stream. The watermark bit rate can characterize the watermark embedding strength. The higher the watermark bit rate, the higher the watermark embedding strength and the higher the watermark security, but the higher the computing power required.

[0114] In some embodiments of this application, the watermark embedding strategy may further include a watermark embedding interval, which indicates the frequency band in the second audio that needs to be embedded with a watermark. According to the Nyquist sampling theorem, the audio frequency range is half the sampling rate, that is, audio sampled at 24000Hz has a frequency range of 0~12000Hz; the human ear does not perceive low frequencies, such as 100Hz–1500Hz, very clearly, therefore, the frequency band in the watermark embedding interval can be in the low frequency range of 100Hz–1500Hz.

[0115] In some embodiments of this application, the watermark embedding strategy may further include a watermark encoding method. Watermark encoding methods can be divided into two types: binary encoding and redundant encoding. Redundant encoding refers to further processing the binary bitstream watermark using check codes such as Bose-Chaudhuri-Hocquenghem Code (BCH), Low-Density Parity-Check Code (LDPC), and parity check codes after obtaining the binary bitstream watermark through binary encoding, thereby enhancing the watermark's resistance to tampering.

[0116] In some embodiments of this application, the computing power of an electronic device can be determined based on the device's state information, which may include hardware state information and software state information. Hardware state information includes information reflecting the hardware status of the electronic device, such as CPU utilization, GPU utilization, DSP utilization, device temperature, and battery level. Software state information may include information reflecting the business scenario in which the electronic device is operating, such as the current application status.

[0117] In some embodiments of this application, different watermark embedding strategies can be adopted for different computing power. Step 140 above may include the following steps 1401-1403.

[0118] Step 1401. Obtain the status information of the electronic device, which includes hardware status information and software status information.

[0119] In some embodiments of this application, the electronic device may include a computing power monitoring module, which can obtain hardware status information such as CPU utilization, GPU utilization, DSP utilization, electronic device temperature, and battery level, as well as software status information such as applications in operation in real time.

[0120] Step 1402. Determine the computing power level of the electronic device based on the status information.

[0121] In some embodiments of this application, information such as hardware occupancy rate, battery level, and business scenario of an electronic device can be determined based on the acquired status information, and then the computing power level of the electronic device can be determined based on the information such as hardware occupancy rate, battery level, and business scenario.

[0122] In some embodiments of this application, the computing power level of the electronic device can be divided into high level, medium level and low level, wherein the computing power level of the high level is higher than that of the medium level, and the computing power level of the medium level is higher than that of the low level.

[0123] In some embodiments of this application, step 1402 above can determine the computing power level of the electronic device based on the following rules: If the electronic device is determined to be in the first scenario based on the status information, or if the hardware occupancy rate of the electronic device is less than the first occupancy rate threshold, the computing power level of the electronic device is determined to be high level. If, based on status information, it is determined that the electronic device's battery level is below the battery threshold, or the electronic device is in the second scenario, or the electronic device's hardware occupancy rate is greater than the second occupancy rate threshold, then the computing power level of the electronic device is determined to be low. If, based on status information, it is determined that the hardware occupancy rate of an electronic device is greater than or equal to a first occupancy rate threshold and less than or equal to a second occupancy rate threshold, the computing power level of the electronic device is determined to be medium.

[0124] In some embodiments of this application, the first scenario may include, but is not limited to, business scenarios with high requirements for audio security, such as recording and evidence storage.

[0125] In some embodiments of this application, the second scenario may include, but is not limited to, business scenarios with low requirements for audio security, such as phone calls and voice assistants.

[0126] In some embodiments of this application, the power threshold, the first occupancy threshold, and the second occupancy threshold can be set according to actual needs. For example, the power threshold can be set to 40%, the first occupancy threshold can be set to 60%, and the second occupancy threshold can be set to 80%, without specific limitations.

[0127] Step 1403. Based on the mapping relationship between computing power level and watermark embedding strategy, determine the watermark embedding strategy corresponding to the computing power level of the electronic device as the watermark embedding strategy that matches the computing power of the electronic device; wherein, the higher the computing power level, the higher the watermark bit rate included in the corresponding watermark embedding strategy.

[0128] In some embodiments of this application, different watermark bit rates are used in different watermark embedding strategies; the higher the computing power level, the higher the watermark bit rate in the corresponding watermark embedding strategy. Thus, by dynamically adjusting the watermark embedding strategy based on the computing power level of the electronic device, energy consumption can be effectively reduced while ensuring the robustness of watermark embedding.

[0129] In some embodiments of this application, when the computing power level of the electronic device is high, the first watermark embedding strategy corresponding to the high level is determined as a watermark embedding strategy that matches the computing power of the electronic device. The first watermark embedding strategy is used to instruct the encoding of the first information using redundant coding to obtain a first watermark; the first watermark is written into a first watermark embedding interval at a first bit rate; wherein the first watermark embedding interval includes the frequency band in the second audio corresponding to the first bit rate.

[0130] In some embodiments of this application, when the computing power level of the electronic device is medium, the second watermark embedding strategy is determined to be a watermark embedding strategy that matches the computing power of the electronic device. The second watermark embedding strategy is used to indicate that the second watermark information is encoded using binary encoding to obtain the second watermark. The second watermark is written into the second watermark embedding interval according to the second bit rate. The second watermark embedding interval includes the frequency band in the second audio corresponding to the second bit rate.

[0131] In some embodiments of this application, when the computing power level of the electronic device is low, the third watermark embedding strategy is determined to be a watermark embedding strategy that matches the computing power of the electronic device. The third watermark embedding strategy is used to indicate that the third watermark information is binary encoded to obtain the third watermark; the third watermark is written into the third watermark embedding interval according to the third bit rate; wherein, the third watermark embedding interval includes the frequency band in the second audio corresponding to the third bit rate.

[0132] The first bit rate is greater than the second bit rate, and the second bit rate is greater than the third bit rate. The values ​​of the first, second, and third bit rates can be set according to actual needs. For example, the first bit rate can be set to 30 bits, the second bit rate can be set to 16 bits, and the third bit rate can be set to 8 bits.

[0133] In some embodiments of this application, the first watermark information may be more complex than the second and third watermark information. For example, the first watermark information may include user identity information, device information, time, geographical location, etc., while the second and third watermark information may include user identity information, device information, time, etc.

[0134] In some embodiments of this application, different bit rates can correspond to different watermark embedding intervals, with higher bit rates corresponding to more frequency bands in the watermark embedding interval. A mapping relationship between bit rate and frequency band can be preset, and the watermark embedding interval corresponding to the bit rate can be determined based on this mapping relationship. For example, the watermark embedding interval freqs = [100, 200, 300, ..., 800] for 8 bits, [100, 150, 200, 250, ..., 750] for 16 bits, and [100, 140, 180, 220, ..., 740, ..., 1500] for 32 bits.

[0135] Accordingly, after step 140 above, step 130 above can be specifically implemented as follows: using the end-side sound replication model, a second audio is generated based on the first audio and the acquired text, and during the generation of the second audio, a watermark is added to the second audio based on the watermark embedding strategy using the watermark embedding module.

[0136] For example, the first watermark information includes user identity information, device information, time, geographical location, etc., with a first bit rate of 32 bits and a first watermark embedding interval freqs = [100, 140, 180, 220,…,740,…,1500]. Based on this, when writing the watermark to the second audio based on the first watermark embedding strategy, the first watermark information is encoded into a 32-bit bitstream, and parity redundancy encoding is applied, such as... Figure 4 Every 8 bits, 0 or 1 is inserted sequentially to form a 36-bit bitstream as the first watermark. For example, the first watermark bit_stream_meidum = [1, 1, 0, 1, 0, 0, 1, 0, 0, 1, 0, 1, 1, 0, 1, 1, 0, 0, 0, 1, 1, 1, 0, 0, 1, 0, 1, 1, 0, 1, 1, 1]. The frequencies freqs = [100, 140, 180, 220, ..., 740, ..., 1500] in the second audio are selected. The bit values ​​bit_stream_low[i] in the first watermark correspond one-to-one with freqs. When bit_stream_low[i] = 0, the time domain amplitude A corresponding to the audio with frequency freqs[i] in the second audio is reduced by a*A. When bit_stream_low[i] = 0, the time domain amplitude A corresponding to the audio with frequency freqs[i] in the second audio is reduced by a*A. 1. The time-domain amplitude A corresponding to the audio with frequency freqs[i] in the second audio is increased by a*A.

[0137] For example, the second watermark information includes user identity information, device information, time, etc., the second bit rate is 16 bits, and the second watermark embedding interval freqs = [100, 150, 200, 250,…,750]. Based on this, when writing a watermark to the second audio based on the second watermark embedding strategy, the second watermark information is encoded into a 16-bit bitstream as the second watermark, such as the second watermark bit_stream_meidum=[1,0,1,0,0,0,1,1,1,0,1,0,0,1,0,0]. Audio frequencies freqs = [100, 150, 200, 250,…,750] in the second audio are selected, and the bit values ​​in the second watermark correspond one-to-one with freqs. When the bit value bit_stream_low[i] = 0, the time domain amplitude A corresponding to the audio with frequency freqs[i] in the second audio is subtracted by a*A; when the bit value bit_stream_low[i] = 1, the time domain amplitude A corresponding to the audio with frequency freqs[i] in the second audio is added by a*A.

[0138] For example, the third watermark information includes user identity information, device information, time, etc., the third bit rate is 8 bits, and the third watermark embedding interval freqs = [100, 200, 300, ..., 800]. Based on this, when writing a watermark to the second audio based on the third watermark embedding strategy, the third watermark information is encoded into an 8-bit bitstream as the third watermark, such as the third watermark bit_stream_meidum=[1,0,1,0,0,0,1,1]. Audio frequencies freqs = [100,200, 300,…,800] in the second audio are selected, and the bit values ​​in the third watermark correspond one-to-one with freqs. When the bit value bit_stream_low[i] = 0, the time domain amplitude A corresponding to the audio frequency freqs[i] in the second audio is subtracted by a*A; when the bit value bit_stream_low[i] = 1, the time domain amplitude A corresponding to the audio frequency freqs[i] in the second audio is added by a*A.

[0139] Based on the above approach, a low-latency watermark embedding method is proposed. The watermark embedding strategy is dynamically adjusted based on the computing power of electronic devices. While ensuring the robustness of watermark embedding, energy consumption is effectively reduced, enabling the edge sound replication model to achieve more reliable operation while balancing performance and security.

[0140] The audio processing method provided in this application can be executed by an electronic device. This application uses an electronic device executing the audio processing method as an example to illustrate the electronic device provided in this application.

[0141] See Figure 5 The diagram below illustrates an electronic device 500 provided in some embodiments of this application. The electronic device 500 is equipped with an end-side forgery detection model, such as... Figure 5 As shown, the electronic device 500 includes: Acquisition module 501 is used to acquire the first audio file to be processed; The forgery detection module 502 is used to perform forgery detection on the first audio through the terminal forgery detection model and obtain the detection result. The detection result is used to indicate whether the first audio is forged audio. The terminal forgery detection model is a lightweight model obtained by compressing the large cloud forgery detection model.

[0142] In this embodiment, by performing forgery detection on the first audio file to be processed, attackers can be prevented from using synthesized speaker voices to deceive and perform voice imitation or other processing. Furthermore, by using an edge-side forgery detection model deployed locally on the electronic device to perform forgery detection on the first audio file, there is no need to upload the first audio file to the cloud. This avoids the latency caused by uploading audio to the cloud, improves the real-time performance of audio processing, and also avoids the risks of theft or misuse that would result from uploading audio to the cloud, thus enhancing the security of audio processing.

[0143] In some embodiments of this application, the generation method of the end-side forgery detection model includes: Build a student model; The cloud-based forgery detection model is used as the teacher model for the student model to train the student model and obtain the initial forgery detection model. The parameters of the initial forgery detection model are then sparsified to obtain the edge-side forgery detection model.

[0144] In some embodiments of this application, the electronic device 500 further deploys an end-side sound replication model, and the electronic device 500 also includes: The audio replication module is used to generate a second audio based on the first audio and the acquired text, using an edge-side audio replication model, when the detection result indicates that the first audio is not fake audio.

[0145] In some embodiments of this application, the electronic device 500 is further deployed with an end-side sound replication model, which includes a watermark embedding module. The watermark embedding module is used to write a watermark into the generated audio during the process of the end-side sound replication model generating audio. Electronic devices 500 also include: The audio replication module is used to generate a second audio based on the first audio and the acquired text using an edge-side audio replication model when the detection result indicates that the first audio is not fake audio. During the generation of the second audio, a watermark is added to the second audio using a watermark embedding module.

[0146] In some embodiments of this application, the electronic device 500 is further deployed with an end-side audio replication model, which includes a watermark embedding module. The watermark embedding module is used to write a watermark into the generated audio during the process of the end-side audio replication model generating audio. The watermark is a binary bit stream. Electronic devices 500 also include: The strategy determination module is used for The system acquires the status information of the electronic device, including hardware and software status information; based on the status information, it determines the computing power level of the electronic device; based on the mapping relationship between the computing power level and the watermark embedding strategy, it determines the watermark embedding strategy corresponding to the computing power level of the electronic device as the watermark embedding strategy that matches the computing power of the electronic device; wherein, the higher the computing power level, the higher the watermark bit rate included in the corresponding watermark embedding strategy; the sound replication module is used to generate a second audio based on the first audio and the acquired text through the edge sound replication model when the detection result indicates that the first audio is not fake audio, and adds a watermark to the second audio through the watermark embedding module based on the watermark embedding strategy during the generation of the second audio.

[0147] The electronic device in this application embodiment can be a terminal, or it can be any other device besides a terminal. For example, the electronic device can be a mobile phone, tablet computer, laptop computer, handheld computer, in-vehicle electronic device, mobile internet device (MID), augmented reality (AR) / virtual reality (VR) device, robot, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. It can also be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc. This application embodiment does not make specific limitations.

[0148] The electronic device in this application embodiment can be a device with an operating system. The operating system can be Android, iOS, or other possible operating systems; this application embodiment does not specifically limit the specific operating system.

[0149] The electronic device provided in this application embodiment can achieve... Figures 1 to 4The various processes implemented in the method implementation examples will not be described again here to avoid repetition.

[0150] Optionally, such as Figure 6 As shown, this application embodiment also provides an electronic device 600, including a processor 601 and a memory 602. The memory 602 stores a program or instructions that can run on the processor 601. When the program or instructions are executed by the processor 601, they implement the various steps of the above-described audio processing method embodiment and can achieve the same technical effect. To avoid repetition, they will not be described again here.

[0151] It should be noted that the electronic devices in the embodiments of this application include the mobile electronic devices and non-mobile electronic devices described above.

[0152] Figure 7 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this application.

[0153] The electronic device 700 includes, but is not limited to, components such as: radio frequency unit 701, network module 702, audio output unit 703, input unit 704, sensor 705, display unit 706, user input unit 707, interface unit 708, memory 709, and processor 710.

[0154] Those skilled in the art will understand that the electronic device 700 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 710 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 7 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0155] The processor 710 is used to acquire the first audio to be processed; The processor 710 is used to perform forgery detection on the first audio through the edge forgery detection model and obtain the detection result. The detection result is used to indicate whether the first audio is forged audio. The edge forgery detection model is a lightweight model obtained by compressing the large cloud forgery detection model.

[0156] In this embodiment, by performing forgery detection on the first audio file to be processed, attackers can be prevented from using synthesized speaker voices to deceive and perform voice imitation or other processing. Furthermore, by using an edge-side forgery detection model deployed locally on the electronic device to perform forgery detection on the first audio file, there is no need to upload the first audio file to the cloud. This avoids the latency caused by uploading audio to the cloud, improves the real-time performance of audio processing, and also avoids the risks of theft or misuse that would result from uploading audio to the cloud, thus enhancing the security of audio processing.

[0157] In some embodiments of this application, the generation method of the end-side forgery detection model includes: Build a student model; The cloud-based forgery detection model is used as the teacher model for the student model to train the student model and obtain the initial forgery detection model. The parameters of the initial forgery detection model are then sparsified to obtain the edge-side forgery detection model.

[0158] In some embodiments of this application, the electronic device further deploys an end-side sound replication model, and the processor 710 is further configured to: If the detection results indicate that the first audio is not fake audio, a second audio is generated based on the first audio and the acquired text using an edge-side sound replication model.

[0159] In some embodiments of this application, the electronic device is further deployed with an end-side sound replication model, which includes a watermark embedding module. The watermark embedding module is used to write a watermark into the generated audio during the process of the end-side sound replication model generating audio. The processor 710 is also used to generate a second audio based on the first audio and the acquired text by means of an end-side sound replication model when the detection result indicates that the first audio is not fake audio, and to add a watermark to the second audio by means of a watermark embedding module during the generation of the second audio.

[0160] In some embodiments of this application, the electronic device is further deployed with an end-side audio replication model, which includes a watermark embedding module. The watermark embedding module is used to write a watermark into the generated audio during the process of the end-side audio replication model generating audio. The watermark is a binary bit stream. The processor 710 is also used to acquire status information of the electronic device, including hardware status information and software status information; Based on status information, determine the computing power level of electronic devices; Based on the mapping relationship between computing power level and watermark embedding strategy, the watermark embedding strategy corresponding to the computing power level of the electronic device is determined as the watermark embedding strategy that matches the computing power of the electronic device; wherein, the higher the computing power level, the higher the watermark bit rate included in the corresponding watermark embedding strategy; when the detection result indicates that the first audio is not fake audio, the second audio is generated based on the first audio and the acquired text through the edge sound replication model, and during the generation of the second audio, the watermark embedding module adds a watermark to the second audio based on the watermark embedding strategy.

[0161] It should be understood that, in this embodiment, the input unit 704 may include a graphics processing unit (GPU) 7041 and a microphone 7042. The GPU 7041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 706 may include a display panel 7061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, or the like. The user input unit 707 includes at least one of a touch panel 7071 and other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include a touch detection device and a touch controller. Other input devices 7072 may include, but are not limited to, a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), a trackball, a mouse, and a joystick, which will not be described in detail here.

[0162] The memory 709 can be used to store software programs and various data. The memory 709 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback, image playback, etc.). Furthermore, the memory 709 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 709 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0163] Processor 710 may include one or more processing units; optionally, processor 710 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into processor 710.

[0164] This application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the various processes of the above-described audio processing method embodiments and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0165] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0166] This application embodiment also provides a chip, which includes a processor and a communication interface. The communication interface is coupled to the processor. The processor is used to run programs or instructions to implement the various processes of the above-described audio processing method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0167] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0168] This application provides a computer program product, which is stored in a storage medium and executed by at least one processor to implement the various processes of the audio processing method embodiments described above, and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0169] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0170] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0171] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. An audio processing method, characterized in that, Performed by an electronic device equipped with an end-to-end forgery detection model, the method includes: Get the first audio file to be processed; The first audio is subjected to forgery detection using the edge-side forgery detection model to obtain a detection result, which is used to indicate whether the first audio is forged audio; wherein, the edge-side forgery detection model is a lightweight model obtained by compressing the large cloud-based forgery detection model.

2. The method according to claim 1, characterized in that, The generation method of the end-side forgery detection model includes: Build a student model; The cloud-based forgery detection model is used as the teacher model for the student model to train the student model, thus obtaining the initial forgery detection model. The initial forgery detection model is subjected to parameter sparsification to obtain the end-side forgery detection model.

3. The method according to any one of claims 1-2, characterized in that, The electronic device also deploys an end-side sound replication model, and the method further includes: If the detection result indicates that the first audio is not fake audio, a second audio is generated based on the first audio and the acquired text using the end-side sound replication model.

4. The method according to claim 1, characterized in that, The electronic device is also equipped with an end-side sound replication model, which includes a watermark embedding module. The method further includes: If the detection result indicates that the first audio is not fake audio, the second audio is generated based on the first audio and the acquired text using the edge sound replication model. During the generation of the second audio, a watermark is added to the second audio using the watermark embedding module.

5. The method according to claim 1, characterized in that, The electronic device is also deployed with an end-side audio replication model, which includes a watermark embedding module. The watermark embedding module is used to write a watermark into the generated audio during the process of the end-side audio replication model generating audio. The watermark is a binary bit stream. The method further includes: Acquire the status information of the electronic device, the status information including hardware status information and software status information; Based on the status information, the computing power level of the electronic device is determined; Based on the mapping relationship between computing power level and watermark embedding strategy, the watermark embedding strategy corresponding to the computing power level of the electronic device is determined as the watermark embedding strategy that matches the computing power of the electronic device; wherein, the higher the computing power level, the higher the watermark bit rate included in the corresponding watermark embedding strategy. If the detection result indicates that the first audio is not fake audio, the second audio is generated based on the first audio and the acquired text using the edge sound replication model. During the generation of the second audio, a watermark is added to the second audio using the watermark embedding module based on the watermark embedding strategy.

6. An electronic device, characterized in that, The electronic device is equipped with an end-to-end forgery detection model, and the electronic device also includes: The acquisition module is used to acquire the first audio file to be processed. The forgery detection module is used to perform forgery detection on the first audio using the terminal forgery detection model and obtain a detection result. The detection result is used to indicate whether the first audio is forged audio. The terminal forgery detection model is a lightweight model obtained by compressing the large cloud-based forgery detection model.

7. The electronic device according to claim 6, characterized in that, The generation method of the end-side forgery detection model includes: Build a student model; The cloud-based forgery detection model is used as the teacher model for the student model to train the student model and obtain an initial forgery detection model; the parameters of the initial forgery detection model are then sparsified to obtain the edge-side forgery detection model.

8. The electronic device according to any one of claims 6-7, characterized in that, The electronic device also deploys an end-side sound replication model, and the electronic device further includes: The audio replication module is used to generate a second audio based on the first audio and the acquired text using the end-side audio replication model, when the detection result indicates that the first audio is not fake audio.

9. The electronic device according to claim 6, characterized in that, The electronic device is also equipped with an end-side sound replication model, which includes a watermark embedding module. The electronic device also includes: The sound replication module is used to generate a second audio based on the first audio and the acquired text using the end-side sound replication model when the detection result indicates that the first audio is not fake audio, and to add a watermark to the second audio using the watermark embedding module during the generation of the second audio.

10. The electronic device according to claim 6, characterized in that, The electronic device is also deployed with an end-side audio replication model, which includes a watermark embedding module. The watermark embedding module is used to write a watermark into the generated audio during the process of the end-side audio replication model generating audio. The watermark is a binary bit stream. The electronic device also includes: A strategy determination module is used to acquire the status information of the electronic device, including hardware status information and software status information; based on the status information, determine the computing power level of the electronic device; based on the mapping relationship between the computing power level and the watermark embedding strategy, determine the watermark embedding strategy corresponding to the computing power level of the electronic device as the watermark embedding strategy that matches the computing power of the electronic device; wherein, the higher the computing power level, the higher the watermark bit rate included in the corresponding watermark embedding strategy. The audio replication module is used to generate a second audio based on the first audio and the acquired text using the edge audio replication model when the detection result indicates that the first audio is not fake audio. During the generation of the second audio, the watermark embedding module adds a watermark to the second audio based on the watermark embedding strategy.

11. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a program or instructions that can run on the processor, the program or instructions being executed by the processor to implement the steps of the audio processing method as described in any one of claims 1-5.

Images