Voice confrontation sample generation method and apparatus, and electronic device

Abstract

The invention provides a voice confrontation sample generation method and device and electronic equipment, and relates to the technical field of voice processing, and the method comprises the steps: receiving an input audio signal, coding the audio signal in a potential space, and obtaining a potential feature code; selecting a disturbance from the constructed migratable disturbance set and adding the disturbance to the potential feature code to obtain a potential feature code after disturbance; and decoding the disturbed potential feature codes to obtain a voice confrontation sample. By adopting the voice confrontation sample generation method and device and the electronic equipment, the problem that the audio quality is reduced due to an existing voice confrontation sample generation method is solved.

Description

Technical Field

[0001] The present application relates to the field of voice security technology, and more specifically, to a method, device and electronic device for generating voice adversarial samples. Background Art

[0002] With the rapid development of artificial intelligence technology, Automatic Speech Recognition (ASR) systems can use deep learning technology to efficiently convert speech into text. This technology has been widely used in industries such as financial services, healthcare, and customer service, improving work efficiency and accuracy. However, this widespread application also brings new challenges, especially in terms of privacy protection. Therefore, how to effectively protect personal privacy while ensuring the quality of voice communication has become an urgent problem to be solved.

[0003] However, existing speech processing methods usually add universal adversarial perturbations to the audio waveform to protect privacy, so that the automatic speech recognition system cannot accurately recognize the speech content. However, this method often leads to a serious degradation of audio quality and introduces a lot of noise, thus affecting normal voice communication. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a method, device and electronic device for generating speech adversarial samples to solve the problem of audio quality degradation caused by existing speech adversarial sample generation methods.

[0005] In a first aspect, an embodiment of the present application provides a method for generating a speech adversarial sample, comprising: Receiving an input current audio signal, encoding the current audio signal in a latent space, and obtaining a current latent feature code; A perturbation is selected from the constructed transferable perturbation set and added to the current potential feature code to obtain the current potential feature code after the perturbation. The perturbations in the transferable perturbation set are obtained by training in the latent space. The current latent feature code after perturbation is decoded to obtain the current speech adversarial sample.

[0006] In an optional implementation, encoding the current audio signal in a latent space to obtain a current latent feature code includes: inputting the current audio signal into a variational autoencoder for latent space encoding to obtain the current latent feature code.

[0007] In an optional embodiment, before receiving the input current audio signal, it also includes: for each initial disturbance, iteratively optimizing the initial disturbance with the minimum audio loss as the optimization goal to obtain an optimized transferable disturbance, the audio loss includes a first audio loss that characterizes the loss between the first potential feature code corresponding to the target audio and the disturbance; based on multiple optimized transferable disturbances, constructing a transferable disturbance set.

[0008] In an optional embodiment, the initial disturbance with the minimum audio loss as the optimization goal is iteratively optimized to obtain an optimized transferable disturbance, including: for this round of optimization process, based on the first potential feature encoding and the previous round of disturbance, determining the first audio loss of this round; based on the original audio adversarial sample of this round, the target audio and the previous round of disturbance, determining the second audio loss of this round; based on the first audio loss of this round and the previous round of disturbance, determining the disturbance of this round, so as to use the disturbance of this round to perform subsequent optimization processes until the optimized transferable disturbance is obtained.

[0009] In an optional embodiment, the original audio adversarial sample of this round is determined by the following processing: the original audio signal of this round is selected from the original audio signal set, and the original audio signal of this round is encoded in the latent space to obtain the second latent feature code of this round; the disturbance of the previous round is added to the second latent feature code of this round in the latent space to generate the disturbed second latent feature code of this round; after decoding the disturbed second latent feature code of this round, the original audio adversarial sample of this round is obtained.

[0010] In an optional embodiment, the method also includes: determining, for this round of optimization process, a first parameter value corresponding to the noise parameter and a second parameter value corresponding to the acoustic environment parameter; adding the first parameter value to the second potential feature code of this round after disturbance, and decoding the second potential feature code of this round after disturbance to obtain the original audio adversarial sample of this round; determining the second audio loss of this round based on the original audio adversarial sample of this round, the second parameter value corresponding to the acoustic environment parameter and the target audio.

[0011] In an optional embodiment, based on the original audio adversarial sample of this round, the second parameter value corresponding to the acoustic environment parameter and the target audio, the loss of the second audio of this round is determined, including: determining the convolution result of the second parameter value and the adversarial sample of the original audio of this round; substituting the convolution result and the target audio into a preset loss function to determine the loss of the second audio of this round.

[0012] In an optional implementation, the current latent feature code after the disturbance is decoded to obtain the current speech adversarial sample, including: using a decoder to reconstruct the current latent feature code after the disturbance to obtain the current speech adversarial sample.

[0013] In a second aspect, an embodiment of the present application further provides a device for generating a speech adversarial sample, the device comprising: The encoding module is used to receive the input current audio signal, encode the current audio signal in the latent space, and obtain the current latent feature code; A disturbance adding module is used to select a disturbance from the constructed transferable disturbance set and add it to the current potential feature code to obtain the current potential feature code after the disturbance. The disturbance in the transferable disturbance set is obtained by training in the latent space. The decoding module is used to decode the current potential feature encoding after the disturbance to obtain the current speech adversarial sample.

[0014] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the memory communicate through the bus, and when the machine-readable instructions are executed by the processor, the steps of the speech adversarial sample generation method as described above are performed.

[0015] The embodiments of the present application bring the following beneficial effects: The embodiments of the present application provide a method, device and electronic device for generating speech adversarial samples, which can encode the current audio signal in a latent space and apply disturbance to the current latent feature coding after disturbance to transfer the disturbance to the latent space, thereby avoiding the problem of audio quality degradation caused by introducing noise into the original audio data space. Compared with the speech adversarial sample generation method in the prior art, the problem of audio quality degradation caused by adding disturbance to speech is solved.

[0016] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A flow chart of a method for generating speech adversarial samples provided in an embodiment of the present application is shown; Figure 2 A flow chart of a single-round disturbance optimization method provided in an embodiment of the present application is shown; Figure 3A flow chart of a method for determining a disturbance in this round provided by an embodiment of the present application is shown; Figure 4 A schematic diagram of the structure of a speech adversarial sample generating device provided in an embodiment of the present application is shown; Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0019] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application usually described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, each other embodiment obtained by those skilled in the art without making creative work belongs to the scope of protection of the present application.

[0020] It is worth noting that before the present application was filed, with the rapid development of artificial intelligence technology, the Automatic Speech Recognition (ASR) system was able to use deep learning technology to efficiently convert speech into text. This technology has been widely used in industries such as financial services, healthcare, and customer service, improving work efficiency and accuracy. For example, in the financial services industry, automatic speech recognition technology is used for voice monitoring to enhance the reliability and efficiency of compliance management. However, this wide application also brings new challenges, especially in terms of privacy protection. Indiscriminate large-scale voice monitoring has aroused public concerns about the leakage of personal privacy, especially in the context of widespread monitoring of personal phone calls and Internet data on a global scale. Therefore, how to effectively protect personal privacy while ensuring the quality of voice communications has become an urgent problem to be solved.

[0021] Existing speech processing methods attempt to protect privacy by adding universal adversarial perturbations to the audio waveform, making it impossible for the automatic speech recognition system to accurately recognize the speech content. However, this method often leads to a serious degradation of audio quality and introduces a lot of noise, thus affecting normal voice communication. At the same time, existing technologies usually require iterative optimization of specific voice inputs, and different iterative optimizations are required for different voices, which cannot meet the needs of real-time voice communication.

[0022] In addition, since users cannot know the specific monitoring methods used by the listener, the specific ASR model is also unknown to users. This requires that the anti-disturbance must be effective for different ASR models. However, existing technologies are usually designed for a single model and are difficult to adapt to different ASR systems.

[0023] Based on this, an embodiment of the present application provides a method for generating speech adversarial samples to improve the audio quality in the process of adding disturbances to speech, as well as the adaptability to different ASR systems.

[0024] See also Figure 1 , Figure 1 This is a flow chart of a method for generating speech adversarial samples provided in an embodiment of the present application. Figure 1 As shown, the method for generating speech adversarial samples provided in the embodiment of the present application includes: Step S101, receiving an input current audio signal, encoding the current audio signal in a latent space, and obtaining a current latent feature code; Step S102, selecting a disturbance from the constructed transferable disturbance set and adding it to the current potential feature code to obtain the current potential feature code after the disturbance; Step S103, decoding the current latent feature code after disturbance to obtain the current speech adversarial sample.

[0025] The speech adversarial sample generation method provided in the embodiment of the present application can encode the current audio signal in the latent space and apply disturbance to the current latent feature encoding after the disturbance to transfer the disturbance to the latent space, thereby avoiding the problem of audio quality degradation caused by introducing noise into the original audio data space, and solving the problem of audio quality degradation caused by adding disturbance to the speech.

[0026] To facilitate understanding of this embodiment, the above exemplary steps provided in the embodiment of the present application are described below by taking the application of the speech adversarial sample generation method to a terminal device as an example.

[0027] In step S101, an input current audio signal is received, and the current audio signal is encoded in a latent space to obtain a current latent feature code.

[0028] In this step, the current audio signal may refer to an audio signal currently received by the terminal device, and the current audio signal may be an audio signal input by a user through a microphone.

[0029] Latent space refers to an abstract high-dimensional space in audio processing, where each point corresponds to a potential audio representation or feature. Latent space is not only used to reduce the dimension of data, but also to capture and express the essential characteristics of audio signals.

[0030] In an embodiment of the present application, when a user inputs voice into a terminal device, the terminal device receives a current audio signal input by the user through a microphone of the terminal device, and inputs the current audio signal into a variational autoencoder or autoencoder for latent space encoding to obtain a current latent feature code.

[0031] Among them, variational autoencoder can refer to VAE (Variational Autoencoder). In the latent space, the current audio signal input is compressed into a lower dimensional representation after VAE encoding, that is, a point in the latent space. This low dimensional representation contains the key features of the input audio, that is, the current latent feature code.

[0032] In step S102, a perturbation is selected from the constructed transferable perturbation set and added to the current potential feature code to obtain the current potential feature code after the perturbation.

[0033] In this step, the perturbations in the transferable perturbation set are obtained by training in the latent space, and the transferable perturbation set is a set of multiple transferable perturbations that are constructed.

[0034] Transferable perturbations can refer to transferable universal adversarial perturbations, each of which can be added to any audio signal to achieve real-time addition of perturbations. Moreover, the perturbations are added in the latent space instead of directly acting on the audio input, which can maintain the audio semantics in voice communication and significantly improve the audio quality.

[0035] In addition, each transferable perturbation is generated based on the target feature adaptive process, so that each transferable perturbation can learn the robust features of the target text and enhance its transferability, so that it is still effective when facing unknown ASR models, thus meeting the requirement of ASR model independence.

[0036] In an embodiment of the present application, a set of migratable disturbances can be stored in a database. After receiving a current audio signal input by a user, a migratable disturbance is randomly selected from the database and added to the current potential feature code corresponding to the current audio signal, thereby obtaining the current potential feature code after disturbance.

[0037] In step S103, the current latent feature code after disturbance is decoded to obtain the current speech adversarial sample.

[0038] In this step, the decoder can be used to reconstruct the current latent feature encoding after disturbance to obtain the current speech adversarial sample.

[0039] In the embodiment of the present application, the current audio adversarial sample is protected audio, which has strong anti-attack capability after disturbance addition processing. Then, the protected audio is transmitted to the virtual microphone of the terminal device, and then the protected audio signal is read from the virtual microphone through the set downstream communication software to ensure that the downstream application receives the protected audio, thereby protecting the user's voice privacy.

[0040] In one example, before applying the speech adversarial sample generation method provided by the embodiment of the present application, it is also necessary to construct a transferable perturbation set, which is constructed based on multiple optimized transferable perturbations, and each optimized transferable perturbation corresponds to an initial perturbation. Here, multiple initial perturbations can be obtained by a random initialization method, and the iterative optimization process of each initial perturbation is the same, and the optimized transferable perturbation corresponding to the initial perturbation is obtained by iteratively optimizing the initial perturbation with the minimum audio loss as the optimization target. Among them, the audio loss includes a first audio loss that characterizes the loss between the first potential feature code corresponding to the target audio and the perturbation, and a second audio loss that characterizes the speech recognition loss.

[0041] The following takes a single initial disturbance as an example to introduce in detail the process of obtaining the optimized transferable disturbance corresponding to the initial disturbance.

[0042] First, the target audio corresponding to the initial disturbance is determined.

[0043] The target audio is a fixed audio selected by experimental methods, so that the initial perturbation can learn the potential features of the target audio through iterative optimization. Each initial perturbation corresponds to a target audio. During multiple rounds of iterative optimization for a certain initial perturbation, the target audio remains unchanged. Here, the text corresponding to the target audio is called the target text, and the target text is denoted as: t.

[0044] The target audio is encoded through VAE to obtain the first latent feature code corresponding to the target audio, thereby realizing the conversion of the target audio into a low-dimensional latent space representation. The first latent feature code is recorded as: .

[0045] Then, the initial disturbance is iteratively optimized using the first latent feature encoding to obtain an optimized transferable disturbance corresponding to the initial disturbance. The iterative optimization process is a multi-round optimization process.

[0046] Refer to the following Figure 2 Let’s introduce the single-round iterative optimization process in detail.

[0047] Figure 2 A flow chart of a single-round disturbance optimization method provided in an embodiment of the present application is shown. Figure 2As shown, the single-round perturbation optimization method includes: Step S201, determining the second potential feature code of this round corresponding to the original audio signal of this round.

[0048] In this round of optimization, an original audio signal is selected from the original audio signal set (training data set) as the original audio signal used in this round of optimization, which is called the original audio signal of this round. The original audio signal of this round is mapped to the latent space through VAE to obtain the latent feature encoding corresponding to the original audio signal of this round, thereby realizing the conversion of the original audio signal into a low-dimensional latent space representation.

[0049] Here, the original audio signal of this round is recorded as: x, and the encoder is recorded as: , the potential feature code corresponding to the original audio signal x in this round is called the second potential feature code in this round, and the second potential feature code in this round is recorded as: , then .

[0050] Step S202, adding disturbance and noise to the second potential feature code of this round to obtain the second potential feature code of this round after disturbance.

[0051] In the latent space, add the previous round perturbation δ and noise parameter p to the second latent feature encoding of this round In the process, the second potential feature code of this round after adding disturbance and noise is generated. The second potential feature code of this round after adding disturbance and noise is recorded as: .

[0052] Alternatively, we can also add only the perturbation δ from the previous round to the second latent feature encoding in the current round in the latent space: In the process, the perturbed second potential feature code of this round is obtained, and the perturbed second potential feature code of this round is decoded to obtain the original audio adversarial sample of this round.

[0053] Step S203, decoding the second latent feature code of this round after adding disturbance and noise to obtain the original audio adversarial sample of this round.

[0054] The decoder is used to decode the second potential feature code of this round after adding disturbance and noise to obtain the original audio adversarial sample of this round. The original audio adversarial sample of this round is recorded as: , the decoder is recorded as: , then .

[0055] Step S204, using the original audio adversarial sample of this round to determine the disturbance of this round.

[0056] In order to improve the mobility of the disturbance, the target feature adaptive method can be used for optimization processing to minimize the cosine similarity loss between the first latent feature encoding corresponding to the target audio and the disturbance, ensuring that the disturbance can be migrated to the target audio, so that the initial disturbance can learn the latent features of the target audio.

[0057] Refer to the following Figure 3 Let’s introduce the determination process of this round of disturbance in detail.

[0058] Figure 3 A flow chart of a method for determining a disturbance in this round provided by an embodiment of the present application is shown. Figure 3 As shown, the method for determining the disturbance of this round includes: Step S2041, determining the total audio loss of this round according to the overall optimization target.

[0059] In one case, when the influence of the physical environment on the disturbance is not included, the overall optimization goal can be expressed as: ; In the above formula, Express expectations, represents the data distribution of natural audio in the physical world, and x represents the expectation in this round The sampling in represents the hyperparameter that adjusts the balance between the two loss terms, f represents the ASR model used to convert audio to text, represents the noise parameter. The minimum value can achieve the effect of allowing the perturbation to learn the potential features of the target audio. The minimum effect can be achieved by interfering with the ASR model.

[0060] In another case, in order to improve the robustness of the perturbation in the real physical environment, the room impulse response (RIR) can be integrated into the perturbation optimization process, that is, the physical environment impact is added to the optimization target. Among them, RIR is used to simulate the environmental effects in the audio propagation process, such as signal attenuation, multipath effect and environmental noise, which will affect the effectiveness of the countermeasure perturbation.

[0061] For example: in the loss function The physical environment influence is added to the noise parameter and the acoustic environment parameter. The noise parameter is expressed as: , the acoustic environment parameters are recorded as: .

[0062] First, the first parameter value corresponding to the noise parameter p and the acoustic environment parameter The corresponding second parameter value is added to the first parameter value to encode the second potential feature of this round after the disturbance to determine the original audio adversarial sample of this round with noise added. The original audio adversarial sample of this round with noise added is recorded as: .

[0063] Then, based on the original audio adversarial sample with noise added , acoustic environment parameter r and target text t corresponding to the target audio, determine the second audio loss of this round. At this time, you can first calculate Corresponding transcript , and determine the second parameter value r and the original audio adversarial sample with noise added The convolution result , substitute the convolution result and target audio into the preset loss function In the above example, the loss of the second audio frequency in this round can be determined. At this time, the loss of the second audio frequency in this round can be expressed as: .

[0064] When the influence of the physical environment on the disturbance is added, the optimization objective can be expressed as: ; In the above formula, R represents the RIR distribution in the physical environment, r represents the parameter value obtained by sampling the RIR distribution once in each round of iterative optimization, represents Gaussian distribution, ⊗ represents convolution operation, which is used to simulate the environmental impact of audio signals during propagation; p represents the Gaussian noise obtained by sampling the Gaussian distribution in each round of iterative optimization, so as to add the Gaussian noise to the disturbance. Represents the variance used to control the noise, which is used to increase the diversity of input and avoid overfitting.

[0065] In addition, the constraints of the overall optimization objective are ,in, It is used to control the disturbance range, which is a set value. At this time, by integrating RIR into the disturbance optimization process, the generated optimized transferable disturbance can have stronger robustness in the physical environment, so as to effectively improve the audio privacy protection ability in different environments.

[0066] Here, the total audio loss of this round includes the first audio loss of this round and the second audio loss of this round. At this time, the first audio loss of this round and the second audio loss of this round need to be calculated respectively.

[0067] For the first audio loss of this round, the calculation method of the first audio loss of this round is the same regardless of whether the influence of the physical environment on the disturbance is added. When calculating the first audio loss of this round, the first potential feature encoding and the last round of disturbance , determine the first audio loss of this round. For example: calculate the cosine similarity loss between the first potential feature code corresponding to the target audio and the previous round of disturbance, and determine the cosine similarity loss as the first audio loss of this round. The first audio loss of this round is recorded as: .

[0068] For the second audio loss of this round, when calculating the second audio loss of this round without adding the influence of the physical environment on the disturbance, the second audio loss of this round can be determined based on the original audio adversarial sample of this round, the target audio and the previous round of disturbance. For example: first determine the original audio adversarial sample of this round Corresponding transcript , the transcribed text The target text t corresponding to the target audio is substituted into the CTC loss function or the cross entropy loss function to obtain the calculation result of the loss function, and the calculation result of the loss function is determined as the second audio loss of this round. The second audio loss of this round is recorded as: .

[0069] At this point, the total audio loss of this round can be determined according to the overall optimization goal, and the total audio loss of this round is recorded as: , .

[0070] For the second audio loss of this round, when adding the influence of the physical environment on the disturbance, the second audio loss of this round can be determined based on the original audio adversarial sample of this round, the target audio and the previous round of disturbance. For example: first determine the original audio adversarial sample of this round Then, determine the convolution result of the second parameter value and the original audio adversarial sample of this round , and the convolution result Corresponding transcript The target text t corresponding to the target audio is substituted into the CTC loss function or the cross entropy loss function to obtain the calculation result of the loss function, and the calculation result of the loss function is determined as the second audio loss of this round. The second audio loss of this round is recorded as: .

[0071] At this point, the total audio loss of this round can be determined according to the overall optimization goal, and the total audio loss of this round is recorded as: , .

[0072] Step S2042: determine the disturbance of this round according to the total audio loss of this round.

[0073] The current round disturbance may be determined based on the current round first audio loss and the current round second audio loss, so as to use the current round disturbance to perform a subsequent optimization process.

[0074] For example, the Projected Gradient Descent (PGD) method is used to iteratively optimize the current round of perturbations. The iterative formula of δ is as follows: ; In the above formula, Represents the learning rate.

[0075] The above formula can be used to determine the current round of disturbance, and then use this round of disturbance to carry out the next round of optimization process. In the next round of optimization process, a new original audio signal will be selected to continue iterative optimization. And so on, until the iteration stop condition is met, the optimization of the initial disturbance is ended, and the optimized transferable disturbance is obtained. Among them, the iteration stop condition can refer to the number of iterations reaching the set number threshold.

[0076] In this way, after multiple rounds of iterative optimization, the optimization goal of minimizing the overall expected loss is achieved, which can make the optimized transferable perturbation universal and effective for different audios.

[0077] The above method can be used to determine the optimized transferable disturbance corresponding to a disturbance. By iteratively optimizing multiple initial disturbances respectively, multiple optimized transferable disturbances can be obtained. A transferable disturbance set is composed of multiple optimized transferable disturbances. In actual use, one can be randomly selected from the transferable disturbance set and added to the current audio signal.

[0078] Based on the same inventive concept, the embodiment of the present application also provides a speech adversarial sample generation device corresponding to the speech adversarial sample generation method. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned speech adversarial sample generation method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0079] See also Figure 4 , Figure 4 This is a schematic diagram of the structure of a speech adversarial sample generation device provided in an embodiment of the present application. Figure 4 As shown in , the speech adversarial sample generating device 300 includes: The encoding module 301 is used to receive an input current audio signal, encode the current audio signal in a latent space, and obtain a current latent feature code; A disturbance adding module 302 is used to select a disturbance from the constructed transferable disturbance set and add it to the current potential feature code to obtain the current potential feature code after disturbance, and the disturbance in the transferable disturbance set is obtained by training in the latent space; The decoding module 303 is used to decode the current potential feature code after disturbance to obtain the current speech adversarial sample.

[0080] See also Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 5 As shown in , the electronic device 400 includes a processor 410 , a memory 420 and a bus 430 .

[0081] The memory 420 stores machine-readable instructions executable by the processor 410. When the electronic device 400 is running, the processor 410 communicates with the memory 420 via the bus 430. When the machine-readable instructions are executed by the processor 410, the above-mentioned Figure 1 The steps of the method for generating speech adversarial samples in the method embodiment shown in the figure, and the specific implementation method thereof can be found in the method embodiment, which will not be described in detail here.

[0082] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the computer program can execute the above-mentioned Figure 1 The steps of the method for generating speech adversarial samples in the method embodiment shown in the figure, and the specific implementation method thereof can be found in the method embodiment, which will not be described in detail here.

[0083] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0084] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0085] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0086] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0087] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0088] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for generating speech adversarial samples, characterized in that: The method comprises: Receiving an input current audio signal, encoding the current audio signal in a latent space, and obtaining a current latent feature code; Selecting a perturbation from the constructed transferable perturbation set and adding it to the current potential feature code to obtain the perturbed current potential feature code, wherein the perturbations in the transferable perturbation set are obtained by training in the latent space; The current potential feature code after the disturbance is decoded to obtain a current speech adversarial sample.

2. The method according to claim 1, characterized in that The encoding of the current audio signal in the latent space to obtain the current latent feature encoding includes: The current audio signal is input into a variational autoencoder for latent space encoding to obtain a current latent feature code.

3. The method according to claim 1, characterized in that: Before receiving the input current audio signal, the method further includes: For each initial disturbance, iteratively optimize the initial disturbance with minimum audio loss as the optimization objective to obtain an optimized transferable disturbance, wherein the audio loss includes a first audio loss representing a loss between a first latent feature code corresponding to the target audio and the disturbance; Based on multiple optimized transferable perturbations, a transferable perturbation set is constructed.

4. The method according to claim 3, characterized in that The iterative optimization of the initial disturbance with the minimum audio loss as the optimization objective to obtain the optimized transferable disturbance includes: For this round of optimization process, based on the first potential feature encoding and the previous round of disturbance, determining the first audio loss of this round; Determine the loss of the second audio of this round based on the original audio adversarial sample of this round, the target audio and the disturbance of the previous round; Based on the first audio loss of the current round, the second audio loss of the current round and the disturbance of the previous round, the disturbance of the current round is determined, so as to use the disturbance of the current round to perform a subsequent optimization process until an optimized transferable disturbance is obtained.

5. The method according to claim 4, characterized in that The original audio adversarial sample of this round is determined by the following processing: Selecting a current round of original audio signals from the original audio signal set, and encoding the current round of original audio signals in a latent space to obtain a current round of second latent feature encoding; Adding the previous round of disturbance to the second potential feature code of this round in the latent space to generate the perturbed second potential feature code of this round; After decoding the perturbed second latent feature code of this round, the original audio adversarial sample of this round is obtained.

6. The method according to claim 5, characterized in that The method further comprises: For this round of optimization process, determine a first parameter value corresponding to the noise parameter and a second parameter value corresponding to the acoustic environment parameter; Adding the first parameter value to the second potential feature code of this round after the disturbance, and decoding the second potential feature code of this round after the disturbance is added to obtain the original audio adversarial sample of this round; The loss of the second audio of this round is determined based on the original audio adversarial sample of this round, the second parameter value corresponding to the acoustic environment parameter and the target audio.

7. The method according to claim 6, characterized in that The determining the loss of the second audio of the current round based on the original audio adversarial sample of the current round, the second parameter value corresponding to the acoustic environment parameter and the target audio includes: Determine a convolution result of the second parameter value and the current round of original audio adversarial sample; Substitute the convolution result and the target audio into a preset loss function to determine the second audio loss of this round.

8. The method according to claim 1, characterized in that The decoding of the current potential feature code after the disturbance to obtain the current speech adversarial sample includes: The decoder is used to reconstruct the current latent feature encoding after the disturbance to obtain the current speech adversarial sample.

9. A speech adversarial sample generation device, characterized in that: include: An encoding module, used for receiving an input current audio signal, encoding the current audio signal in a latent space, and obtaining a current latent feature code; A disturbance adding module, used for selecting a disturbance from the constructed transferable disturbance set and adding it to the current potential feature code to obtain the current potential feature code after disturbance, wherein the disturbance in the transferable disturbance set is obtained by training in the potential space; A decoding module is used to decode the current potential feature code after the disturbance to obtain a current speech adversarial sample.

10. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory via the bus, and the processor executes the machine-readable instructions to perform the steps of the method for generating speech adversarial samples as described in any one of claims 1 to 8.

Images