Adversarial attack method, device and equipment for speech semantic system

Abstract

The application provides an adversarial attack method, device and equipment for a speech semantic system. The method comprises: obtaining to-be-attacked speech information; performing frequency spectrum conversion processing on the to-be-attacked speech information to obtain to-be-attacked frequency spectrum information; inputting the to-be-attacked frequency spectrum information into a pre-trained speech semantic communication network to output a corresponding adversarial attack result; and the loss function of the pre-trained speech semantic communication network is a joint result based on a semantic similarity loss and a connectionist temporal classification loss. Through the adversarial attack processing of the pre-trained speech semantic communication network on the to-be-attacked frequency spectrum information, the problem that the anti-attack and anti-disturbance performance of the speech information in the prior art is weak is avoided; the joint loss function setting strategy based on the semantic similarity loss and the connectionist temporal classification loss realizes accurate generation control of the adversarial attack result, so that the adversarial attack result improves the effective interference on the identification and classification ability of the communication system on the basis of maintaining the speech semantic information.

Description

Technical Field

[0001] This invention belongs to the field of computer and network security technology, and specifically relates to a method, apparatus and device for countering attacks on speech and semantic systems. Background Technology

[0002] In the field of wireless communication, semantic communication has attracted much attention for its efficient information transmission and accurate intent recognition capabilities. However, due to the openness of wireless channels, information is highly susceptible to various forms of interference and attacks during transmission. At the same time, neural networks, as the core technology of semantic communication, are somewhat inadequate in the face of malicious attacks due to their inherent vulnerability.

[0003] The openness of wireless channels means that any individual with the appropriate receiving equipment could potentially intercept transmitted information. This characteristic means that while semantic communication enjoys the convenience of transmission, it also faces the risk of information leakage and tampering. Once an attacker successfully intercepts and modifies the transmitted information, it can not only lead to communication failure but also potentially cause serious security problems. Even more serious is the fact that neural networks, as a key component of semantic communication, are vulnerable to various forms of attack due to their complex internal structure and parameters. Attackers can mislead neural networks with carefully designed input data, causing them to produce incorrect outputs. This type of attack is not only difficult to prevent but also often highly covert, posing a significant challenge to the security of semantic communication systems. Currently, research on the security of semantic communication mainly focuses on the image and text domains, while the security of semantic communication in the audio domain has not received sufficient attention and in-depth exploration. Summary of the Invention

[0004] To address the aforementioned problems in the prior art, this invention provides a method, apparatus, and device for countering attacks on speech semantic systems.

[0005] The technical problem to be solved by this invention is achieved through the following technical solution:

[0006] In a first aspect, the present invention provides an adversarial attack method against speech semantic systems, comprising:

[0007] Obtain the voice information of the opponent;

[0008] The speech information to be challenged is subjected to spectral transformation processing to obtain the spectral information to be challenged.

[0009] The spectrum information to be challenged is input into the pre-trained speech-semantic communication network, which outputs the corresponding adversarial attack results. The loss function of the pre-trained speech-semantic communication network is a joint result of semantic similarity loss and connectionist temporal classification loss.

[0010] Optionally, the speech information to be adversarial is subjected to spectral transformation processing to obtain the spectral information to be adversarial, including:

[0011] The adversarial speech information is processed sequentially by windowing, fast Fourier transform, logarithmic operation, and power normalization to obtain the adversarial spectrum information.

[0012] Optionally, the pre-trained speech and semantic communication network includes, in sequence, a pre-trained channel encoder module, a pre-trained semantic encoder module, a pre-trained feature decoder module, a pre-trained channel decoder module, a fading channel module, and a noise module.

[0013] Optionally, the training process of the pre-trained speech-semantic communication network includes:

[0014] Obtain adversarial speech samples;

[0015] The adversarial speech samples are processed sequentially by windowing, fast Fourier transform, logarithmic operation, and power normalization to obtain the spectrum of the adversarial samples.

[0016] Based on the spectrum of adversarial examples Target sentence y t The loss function is used to iteratively train the initial speech semantic communication network; the initial speech semantic communication network has the same structure as the pre-trained speech semantic communication network.

[0017] The initial speech-semantic communication network that has reached the preset convergence condition is used as the pre-trained speech-semantic communication network.

[0018] Optionally, based on the spectrum of adversarial examples Target sentence y t The loss function is used to iteratively train the initial speech-semantic communication network, including:

[0019] S201, Based on the initial semantic encoder module and initial channel encoder module Adversarial sample spectrum The encoding process is performed sequentially to obtain the semantic sample vector x;

[0020]

[0021] in, Represents the initial semantic encoder module The trainable parameters, β represents the initial channel encoder module. Trainable parameters;

[0022] S202. Based on the fading channel module and the noise module, noise is added to the semantic sample vector x to obtain the semantic noise sample y.

[0023] y = Hx + n;

[0024] Where H represents the coefficients of the linear channel between the transmitter and receiver, and the transmitter consists of the initial semantic encoder module. and initial channel encoder module The receiver is composed of an initial feature decoder module. and the initial channel decoder module Composition; n represents the noise present in the noise module;

[0025] S203, Based on the initial channel decoder module Decode the semantically noisy sample y to obtain the sample probability matrix.

[0026]

[0027] Where θ represents the initial channel decoder module Trainable parameters;

[0028] S204, Initial Feature Decoder Module Greedy decoding rule for sample probability matrix Decoding is performed to obtain the sample text sequence.

[0029]

[0030] S205, along the descent direction of the loss function or the target sentence y t With sample text sequence Repeat steps S201-S204 in the direction of approach to complete the iterative training of the initial speech-semantic communication network.

[0031] Alternatively, the loss function can be expressed as:

[0032]

[0033] in, L represents the connectionist temporal classification loss. s y represents the semantic similarity loss. t Indicates the target sentence. y t This indicates the adversarial sample spectrum. Recognized as y t The process, Let represent the sample probability distribution, β represent the hyperparameter, and loss represent the loss function.

[0034] Alternatively, the connectionist temporal classification loss can be represented as:

[0035]

[0036] in, This represents the connectionist temporal classification loss. Represents the sample probability distribution The output result is y t The likelihood probability at time;

[0037]

[0038] Represents the sample probability distribution The likelihood probability when π is aligned to the bottom, C(·) represents the decoding rule that selects the character with the highest probability score. Let represent the k-th element of the sample probability distribution, and ∏(·) denote the dot product operation.

[0039] Optionally, the semantic similarity loss is expressed as:

[0040]

[0041] Among them, I (·) This indicates the indicator function, and n represents the sample text sequence. The length of y, m represents the target sentence y t Length, This represents the minimum edit distance between n and m.

[0042] Secondly, the present invention provides an adversarial attack device for a speech semantic system, which includes: an acquisition unit, a conversion unit and an attack unit.

[0043] The acquisition unit is used to: acquire the speech information to be challenged;

[0044] The conversion unit is used to: perform spectral conversion processing on the speech information to be countered, and obtain the spectral information to be countered;

[0045] The attack unit is used to: input the spectrum information to be challenged into the pre-trained speech-semantic communication network and output the corresponding adversarial attack result; the loss function of the pre-trained speech-semantic communication network is a joint result of semantic similarity loss and connectionist temporal classification loss.

[0046] Thirdly, the present invention provides an adversarial attack device for a speech semantic system, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the adversarial attack device for a speech semantic system is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the adversarial attack method for a speech semantic system as described in the first aspect above.

[0047] This invention provides a method, apparatus, and device for adversarial attacks against speech semantic systems. The method includes: acquiring speech information to be adversarially attacked; performing spectral transformation processing on the speech information to be adversarially attacked to obtain spectral information to be adversarially attacked; inputting the spectral information to be adversarially attacked into a pre-trained speech semantic communication network, and outputting the corresponding adversarial attack result; the loss function of the pre-trained speech semantic communication network is a joint result based on semantic similarity loss and connectionist temporal classification loss. In this invention, by using a pre-trained speech semantic communication network to perform adversarial attack processing on the spectral information to be adversarially attacked, the problem of weak anti-attack and anti-disturbance performance of speech information in existing technologies is avoided. Furthermore, the strategy of setting the joint loss function based on semantic similarity loss and connectionist temporal classification loss achieves precise generation control of the adversarial attack result, enabling the adversarial attack result to effectively interfere with the recognition and classification capabilities of the communication system while maintaining the speech semantic information.

[0048] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0049] Figure 1 A flowchart illustrating an adversarial attack method for a speech semantic system provided in an embodiment of the present invention;

[0050] Figure 2 The decibel distortion comparison results between the method of the present invention and existing methods provided in the embodiments of the present invention;

[0051] Figure 3 The comparison results of the production time of the method of the present invention and the existing methods provided in the embodiments of the present invention;

[0052] Figure 4 A waveform comparison of the adversarial attack results between the method of the present invention and existing methods provided in the embodiments of the present invention;

[0053] Figure 5 A schematic diagram of a device for countering attacks on speech and semantic systems provided in an embodiment of the present invention;

[0054] Figure 6 This is a schematic diagram of a device for countering attacks on speech and semantic systems, provided as an embodiment of the present invention. Detailed Implementation

[0055] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0056] In order to improve the effectiveness of adversarial attacks in interfering with the identification and classification capabilities of communication systems, this invention provides an adversarial attack method for speech semantic systems. Figure 1 This is a flowchart illustrating an adversarial attack method for speech semantic systems provided in an embodiment of the present invention, as shown below. Figure 1 As shown, it includes:

[0057] S101. Obtain the voice information to be countered.

[0058] S102. Perform spectrum conversion processing on the speech information to be countered to obtain the spectrum information to be countered.

[0059] Optionally, S102 may specifically include:

[0060] The adversarial speech information is processed sequentially by windowing, fast Fourier transform, logarithmic operation, and power normalization to obtain the adversarial spectrum information.

[0061] S103. Input the spectrum information to be challenged into the pre-trained speech semantic communication network and output the corresponding adversarial attack results.

[0062] The loss function of the pre-trained speech semantic communication network is a joint result of semantic similarity loss and connectionist temporal classification loss.

[0063] It should be noted that one application scenario of the method in this embodiment of the invention is that, in a war scenario, it is necessary to give the enemy false command information. However, if the false command information is sent out directly, the spectrum and waveform of the false command information will inevitably differ significantly from those of the real command information, making it very easy for the enemy to detect that it is false information. Therefore, this embodiment of the invention uses a covert pre-trained speech semantic communication network to perform adversarial attack processing on the original speech information (the speech information to be attacked). Since the noise added through the pre-trained speech semantic communication network is relatively subtle, it is not easily detected by the human ear or detection equipment, thereby achieving the deception of the enemy.

[0064] This invention provides an adversarial attack method for speech-semantic systems, comprising: acquiring speech information to be adversarially attacked; performing spectral transformation processing on the speech information to be adversarially attacked to obtain spectral information to be adversarially attacked; inputting the spectral information to be adversarially attacked into a pre-trained speech-semantic communication network and outputting the corresponding adversarial attack result; the loss function of the pre-trained speech-semantic communication network is a joint result based on semantic similarity loss and connectionist temporal classification loss. In this invention, by using a pre-trained speech-semantic communication network to perform adversarial attack processing on the spectral information to be adversarially attacked, the problem of weak anti-attack and anti-disturbance performance of speech information in the prior art is avoided; secondly, the strategy of setting the joint loss function based on semantic similarity loss and connectionist temporal classification loss achieves precise generation control of the adversarial attack result, so that the adversarial attack result improves the effective interference on the recognition and classification capabilities of the communication system while maintaining the speech-semantic information.

[0065] Optionally, the pre-trained speech and semantic communication network includes, in sequence, a pre-trained channel encoder module, a pre-trained semantic encoder module, a pre-trained feature decoder module, a pre-trained channel decoder module, a fading channel module, and a noise module.

[0066] Optionally, the training process of the pre-trained speech-semantic communication network includes:

[0067] Obtain adversarial speech samples;

[0068] The adversarial speech samples are processed sequentially by windowing, fast Fourier transform, logarithmic operation, and power normalization to obtain the spectrum of the adversarial samples.

[0069] Based on the spectrum of adversarial examples Target sentence y t The loss function is used to iteratively train the initial speech semantic communication network; the initial speech semantic communication network has the same structure as the pre-trained speech semantic communication network.

[0070] The initial speech-semantic communication network that has reached the preset convergence condition is used as the pre-trained speech-semantic communication network.

[0071] Optionally, based on the spectrum of adversarial examples Target sentence y t The loss function is used to iteratively train the initial speech-semantic communication network, including:

[0072] S201, Based on the initial semantic encoder module and initial channel encoder module Adversarial sample spectrum The encoding process is performed sequentially to obtain the semantic sample vector x.

[0073]

[0074] in, Represents the initial semantic encoder module The trainable parameters, β represents the initial channel encoder module. Trainable parameters.

[0075] S202. Based on the fading channel module and the noise module, noise is added to the semantic sample vector x to obtain the semantic noise sample y.

[0076] y = Hx + n;

[0077] Where H represents the coefficients of the linear channel between the transmitter and receiver, and the transmitter consists of the initial semantic encoder module. and initial channel encoder module The receiver is composed of an initial feature decoder module. and the initial channel decoder module Composition; n represents the noise present in the noise module.

[0078] It should be noted that in the embodiments of the present invention, N is used. t One transmitting antenna and N r The formula is expressed for each receiving antenna; therefore, semantic noise samples

[0079] S203, Based on the initial channel decoder module Decode the semantically noisy sample y to obtain the sample probability matrix.

[0080]

[0081] Where θ represents the initial channel decoder module Trainable parameters.

[0082] S204, Initial Feature Decoder Module Greedy decoding rule for sample probability matrix Decoding is performed to obtain the sample text sequence.

[0083]

[0084] S205, along the descent direction of the loss function or the target sentence y t With sample text sequence Repeat steps S201-S204 in the direction of approach to complete the iterative training of the initial speech-semantic communication network.

[0085] Alternatively, the loss function can be expressed as:

[0086]

[0087] in, L represents the connectionist temporal classification loss. s y represents the semantic similarity loss. t Indicates the target sentence. y t This indicates the adversarial sample spectrum. Recognized as y t The process, Let represent the sample probability distribution, β represent the hyperparameter, and loss represent the loss function.

[0088] Alternatively, the connectionist temporal classification loss can be represented as:

[0089]

[0090] in, This represents the connectionist temporal classification loss. Represents the sample probability distribution The output result is y t The likelihood probability at time;

[0091]

[0092] Represents the sample probability distribution The likelihood probability when π is aligned to the bottom, C(·) represents the decoding rule that selects the character with the highest probability score. Let represent the k-th element of the sample probability distribution, and ∏(·) denote the dot product operation.

[0093] Optionally, the semantic similarity loss is expressed as:

[0094]

[0095] Among them, I (·) This indicates the indicator function, and n represents the sample text sequence. The length of y, m represents the target sentence y t Length, This represents the minimum edit distance between n and m. For the indicator function I... (·) If the condition within (·) is true, then I (·) The value is 1, otherwise I (·) The value is 0.

[0096] Optionally, in this embodiment of the invention, the termination condition for iterative optimization is that the target attack is successful when the final sample text sequence is the same as the target sentence, and the optimal adversarial attack result is obtained at this time.

[0097] Additionally, it should be noted that in this embodiment of the invention, the parameters of the initial speech-semantic communication network can also be updated using the stochastic gradient descent algorithm to obtain the final pre-trained speech-semantic communication network.

[0098] In this embodiment of the invention, the original speech information (speech information to be challenged) is subjected to adversarial attack processing through a covert pre-trained speech semantic communication network. Since the noise added through the pre-trained speech semantic communication network is relatively subtle, the spectrum and waveform of the adversarial attack result obtained by the adversarial attack are extremely similar to the input speech information to be challenged, causing the enemy to be unable to correctly identify the adversarial attack result, thus achieving the goal of deceiving the enemy.

[0099] To verify the effectiveness of an adversarial attack method targeting speech and semantic systems, simulation experiments were also conducted in this embodiment of the invention.

[0100] 1. Simulation conditions

[0101] The simulation experiments were conducted on an Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz CPU, NVIDIA RTX A4000 GPU, and Ubuntu 20.04 operating system. The experimental datasets used were the public domain speech datasets LJSpeechdataset and LibriSpeech dataset.

[0102] The methods compared in the experiment are as follows:

[0103] The first method is the iterative attack method that generates adversarial examples, referred to as PGD in the experiment. The reference is "Madry A, Makelov A, Schmidt L, et al. Towards Deep Learning Models Resistant to Adversarial Attacks[C] / / International Conference on LearningRepresentations.2018.".

[0104] The second type is an attack method that applies small perturbations based on the gradient of the loss function with respect to the input sample. In the experiment, it is referred to as BIM. The reference is "Kurakin A, Goodfellow IJ, Bengio S. Adversarial examples in the physical world [M] / / Artificial intelligence safety and security. Chapman and Hall / CRC, 2018:99-112".

[0105] The third type is an optimization-based attack method, referred to as C&W in the experiment. The reference is "Carlini N, Wagner D. Towards evaluating the robustness of neural networks[C] / / 2017ieee symposiumon security and privacy(sp).Ieee,2017:39-57.".

[0106] 2. Evaluation Indicators

[0107] This invention will evaluate the method from three perspectives: attack efficiency, attack effectiveness, and attack concealment. The definition of each indicator is as follows:

[0108] (1) Attack efficiency: Attack efficiency is measured by the time it takes for the corresponding method to generate a single adversarial speech sample (adversarial attack result).

[0109] (2) Attack effectiveness: Attack effectiveness is measured by the attack success rate (%), which represents the ratio of the number of successful adversarial attacks against the system to the total.

[0110] (3) Attack concealment: Attack concealment is determined by decibel distortion (dB). x (δ) measures the difference in decibels between the quantized perturbation δ and the original speech signal x, as shown below:

[0111] dB x (δ)=dB(δ)-dB(x);

[0112] 3. Simulation Content

[0113] This invention measures the decibel distortion and production time of adversarial attack results, and compares the method of this invention with the PGD method, BIM method, and C&W method. The results are as follows: Figure 2 and Figure 3 As shown. Figure 2 The following is a comparison of decibel distortion between the method of the present invention and existing methods provided in the embodiments of the present invention. Figure 3 This is a comparison of the production time of the method of the present invention and existing methods provided in this embodiment. T1 to T6 represent the target sentences "play music", "open the door", "send a message", "play music on player", "please open the side door", and "send him a message", respectively. Figure 2 As can be seen, the method of the present invention can generate adversarial attack results with lower perturbation distortion in a shorter time.

[0114] To further illustrate the advanced nature of the method of the present invention, waveforms of adversarial attack results generated using the method of the present invention and the comparative method are also plotted. Figure 4 The waveform comparison results of the adversarial attack results of the method of the present invention and existing methods provided in the embodiments of the present invention are shown. Figure 4 As can be seen, the adversarial attack results generated by the method of the present invention exhibit a high degree of similarity to the original audio in the waveform diagram, and the noise components highlighted in red in the waveform diagram are relatively few. This indicates that the method of the present invention is imperceptible during adversarial processing, further illustrating the advanced nature of the method.

[0115] In addition, experimental evaluations based on the method of this invention were conducted on the public domain speech LJSpeech dataset and the LibriSpeech dataset, and the results are shown in Table 1.

[0116] Table 1 shows the results of adversarial attacks based on the method of this invention on public datasets.

[0117]

[0118]

[0119] In the adversarial attack processing, the speech sample is segmented into 2-second segments, and 2 to 3 words are randomly selected from each segment to form a target phrase. The experimental goal is for the speech-semantic communication system to transcribe the generated adversarial attack result into the target phrase. The method of this invention is evaluated from three perspectives: attack efficiency, attack effectiveness, and attack concealment.

[0120] 1. Attack Efficiency: For short phrases such as "play music" or "open the door," the time cost is less than 11 seconds. In contrast, for long phrases such as "play music on player" or "please open the side door," the time cost is less than 12 seconds. Therefore, the experimental results show that adversarial attack results for specified audio can be generated relatively quickly in both cases, with short target phrases showing higher attack efficiency than long target phrases.

[0121] 2. Attack Effectiveness: The method of this invention achieves an attack success rate of almost 100% against short target phrases. While the attack success rate against long target phrases is slightly lower, it still reaches 96.67%. This demonstrates that the method of this invention has a high attack success rate, enabling semantic communication systems to transmit specified information.

[0122] 3. Attack Stealth: For long target phrases, higher power perturbations are required for a successful attack. Attack samples targeting short sentences can achieve an average relative decibel distortion of -41.45 dB, while for long sentences, it can reach an average of -40.89 dB. The adversarial attack results generated by the method of this invention have such low decibel distortion levels that the adversary would perceive them as natural ambient noise rather than malicious commands, demonstrating the excellent attack stealth of the algorithm proposed in this invention.

[0123] In summary, the adversarial attack method for speech-semantic systems provided by this invention, based on the characteristics of speech signals and the processing flow of communication systems, utilizes a pre-trained speech-semantic communication network to add adversarial perturbations at the source level that are auditorily similar to natural speech and difficult for the human auditory system to detect. Simultaneously, this invention employs a strategy of joint semantic similarity loss and connectionist temporal classification loss, which can precisely control the generation of adversarial attack results. While maintaining the speech-semantic information of the adversarial attack results, it enables the results to effectively interfere with the recognition and classification capabilities of the adversarial communication system. Furthermore, this invention uses a gradient descent algorithm to optimize network model parameters, overcoming the shortcomings of existing methods in terms of insufficient concealment and easy detection by the human ear, thus improving the concealment of the adversarial attack results. This invention addresses the security vulnerabilities of speech-semantic communication systems under adversarial attacks, providing strong support for the design of security hardening and defense strategies for speech-semantic systems.

[0124] The method provided in this embodiment of the invention can be applied to electronic devices. Specifically, the electronic device can be a desktop computer, a portable computer, a smart mobile terminal, a server, etc., and this embodiment of the invention does not limit the application to such devices.

[0125] Based on the same inventive concept, embodiments of the present invention also provide an adversarial attack device for speech semantic systems. Figure 5This is a schematic diagram of a device for countering attacks on speech and semantic systems, provided as an embodiment of the present invention. Figure 5 As shown, it includes: an acquisition unit 501, a conversion unit 502, and an attack unit 503;

[0126] The acquisition unit 501 is used to: acquire the speech information to be countered;

[0127] The conversion unit 502 is used to: perform spectrum conversion processing on the speech information to be countered, and obtain the spectrum information to be countered;

[0128] The attack unit 503 is used to: input the spectrum information to be challenged into the pre-trained speech and semantic communication network and output the corresponding adversarial attack result; the loss function of the pre-trained speech and semantic communication network is the joint result of semantic similarity loss and connectionist temporal classification loss.

[0129] Figure 6 A schematic diagram of a device for countering attacks on a speech and semantic system, provided in an embodiment of the present invention, includes: a processor 710, a storage medium 720, and a bus 730. The storage medium 720 stores machine-readable instructions executable by the processor 710. When the device for countering attacks on a speech and semantic system is running, the processor 710 communicates with the storage medium 720 via the bus 730, and the processor 710 executes the machine-readable instructions to perform the steps of the above-described method embodiment. Specific implementations and technical effects are similar and will not be repeated here.

[0130] The storage medium may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the storage medium may also be at least one storage device located remotely from the aforementioned processor.

[0131] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0132] It should be noted that the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present invention. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the invention.

[0133] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described in this specification.

[0134] Although the invention has been described herein in conjunction with various embodiments, those skilled in the art will understand and implement other variations of the disclosed embodiments by reviewing the accompanying drawings and the disclosure in carrying out the claimed invention. In the description of the invention, the word "comprising" does not exclude other components or steps, "a" or "an" does not exclude a plurality, and "a plurality" means two or more, unless otherwise explicitly specified. Furthermore, while different embodiments may describe certain measures, this does not mean that these measures cannot be combined to produce good results.

[0135] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A method for adversarial attacks against speech semantic systems, characterized in that, include: Obtain the voice information of the opponent; The speech information to be countered is subjected to spectral conversion processing to obtain the spectral information to be countered; The adversarial spectrum information is input into a pre-trained speech-semantic communication network, and the corresponding adversarial attack result is output; the loss function of the pre-trained speech-semantic communication network is a joint result of semantic similarity loss and connectionist temporal classification loss; The training process of the pre-trained speech-semantic communication network includes: Based on the spectrum of adversarial examples Target sentence y t The initial speech-semantic communication network is iteratively trained using a loss function; the initial speech-semantic communication network has the same structure as the pre-trained speech-semantic communication network. The adversarial sample spectrum Target sentence y t The loss function is used to iteratively train the initial speech-semantic communication network, including: S201, Based on the initial semantic encoder module and initial channel encoder module For the adversarial sample spectrum The encoding process is performed sequentially to obtain the semantic sample vector x; in, Represents the initial semantic encoder module The trainable parameters, β represents the initial channel encoder module. Trainable parameters; S202. Based on the fading channel module and the noise module, noise is added to the semantic sample vector x to obtain a semantic noise sample y. y = Hx + n; Where H represents the coefficients of the linear channel between the transmitter and receiver, and the transmitter is defined by the initial semantic encoder module. and initial channel encoder module The receiver is composed of an initial feature decoder module. and the initial channel decoder module Composition; n represents the noise present in the noise module; S203, Based on the initial channel decoder module The semantic noise sample y is decoded to obtain the sample probability matrix. Where θ represents the initial channel decoder module Trainable parameters; S204, Initial Feature Decoder Module And the greedy decoding rule for the sample probability matrix Decoding is performed to obtain the sample text sequence. S205, along the descent direction of the loss function or the target sentence y t With the sample text sequence In the direction of approach, repeat steps S201-S204 to complete the iterative training of the initial speech-semantic communication network.

2. The adversarial attack method against a speech semantic system according to claim 1, characterized in that, The step of performing spectral transformation processing on the speech information to be countered to obtain the spectral information to be countered includes: The speech information to be countered is sequentially processed by windowing, fast Fourier transform, logarithmic operation, and power normalization to obtain the spectrum information to be countered.

3. The adversarial attack method against a speech semantic system according to claim 1, characterized in that, The pre-trained speech and semantic communication network includes, in sequence: a pre-trained channel encoder module, a pre-trained semantic encoder module, a pre-trained feature decoder module, a pre-trained channel decoder module, a fading channel module, and a noise module.

4. The adversarial attack method for a speech semantic system according to claim 1, characterized in that, The training process of the pre-trained speech-semantic communication network includes: Obtain adversarial speech samples; The adversarial speech samples are sequentially processed by windowing, fast Fourier transform, logarithmic operation, and power normalization to obtain the adversarial sample spectrum. Based on the adversarial sample spectrum Target sentence y t The initial speech-semantic communication network is iteratively trained using a loss function; the initial speech-semantic communication network has the same structure as the pre-trained speech-semantic communication network. The initial speech-semantic communication network that reaches the preset convergence condition is used as the pre-trained speech-semantic communication network.

5. The adversarial attack method for a speech semantic system according to claim 1, characterized in that, The loss function is expressed as: in, L represents the connectionist temporal classification loss. s y represents the semantic similarity loss. t Indicates the target sentence. y t This indicates the adversarial sample spectrum. Recognized as y t The process, Let represent the sample probability distribution, β represent the hyperparameter, and loss represent the loss function.

6. The adversarial attack method against a speech semantic system according to claim 5, characterized in that, The connectionist temporal classification loss is expressed as: in, This represents the connectionist temporal classification loss. Represents the sample probability distribution The output result is y t The likelihood probability at time; Represents the sample probability distribution The likelihood probability when π is aligned to the bottom, C(·) represents the decoding rule that selects the character with the highest probability score. Let represent the k-th element of the sample probability distribution, and ∏(·) denote the dot product operation.

7. The adversarial attack method for a speech semantic system according to claim 6, characterized in that, The semantic similarity loss is expressed as: Among them, I (·) This indicates the indicator function, and n represents the sample text sequence. The length of y, m represents the target sentence y t Length, This represents the minimum edit distance between n and m.

8. A device for adversarial attacks against speech and semantic systems, characterized in that, The adversarial attack device against a speech semantic system includes: an acquisition unit, a conversion unit, and an attack unit; The acquisition unit is used to: acquire the voice information to be countered; The conversion unit is used to: perform spectrum conversion processing on the speech information to be countered to obtain the spectrum information to be countered; The attack unit is used to: input the spectrum information to be challenged into a pre-trained speech-semantic communication network and output the corresponding adversarial attack result; the loss function of the pre-trained speech-semantic communication network is a joint result of semantic similarity loss and connectionist temporal classification loss; The training process of the pre-trained speech-semantic communication network includes: Based on the spectrum of adversarial examples Target sentence y t The initial speech-semantic communication network is iteratively trained using a loss function; the initial speech-semantic communication network has the same structure as the pre-trained speech-semantic communication network. The adversarial sample spectrum Target sentence y t The loss function is used to iteratively train the initial speech-semantic communication network, including: Based on the initial semantic encoder module and initial channel encoder module For the adversarial sample spectrum The encoding process is performed sequentially to obtain the semantic sample vector x; in, Represents the initial semantic encoder module The trainable parameters, β represents the initial channel encoder module. Trainable parameters; The semantic sample vector x is subjected to noise addition processing based on the fading channel module and the noise module to obtain the semantic noise sample y; y = Hx + n; Where H represents the coefficients of the linear channel between the transmitter and receiver, and the transmitter is defined by the initial semantic encoder module. and initial channel encoder module The receiver is composed of an initial feature decoder module. and the initial channel decoder module Composition; n represents the noise present in the noise module; Based on the initial channel decoder module The semantic noise sample y is decoded to obtain the sample probability matrix. Where θ represents the initial channel decoder module Trainable parameters; Based on the initial feature decoder module And the greedy decoding rule for the sample probability matrix Decoding is performed to obtain the sample text sequence. Along the descent direction of the loss function or the target sentence y t With the sample text sequence In the direction of approach, repeat steps S201-S204 to complete the iterative training of the initial speech-semantic communication network.

9. A device for adversarial attacks against speech and semantic systems, characterized in that, include: The device includes a processor, a storage medium, and a bus. The storage medium stores machine-readable instructions executable by the processor. When the device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of the method for counterattacking a speech and semantic system as described in any one of claims 1-7.

Images