Audio signal generation method and device, equipment and storage medium thereof
By introducing phase amplitude prediction network and adversarial generation training in the audio signal generation method, the artifact problems in audio signal generation and the problems of phase information ignorance in the prior art are solved, and audio signal generation with higher quality and clarity is achieved.
Patent Information
- Application Number
- CN202411768065.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-03
- Publication Date
- 2025-03-07
AI Technical Summary
Existing neurovocoder technology is prone to artifacts when generating audio signals and ignores phase information, resulting in a decrease in quality and clarity of generated audio, affecting subsequent speech synthesis and user experience.
An audio signal generation method is adopted to obtain the target audio clip and input it to the pre-trained audio feature extraction network for feature extraction. Then, the features are input to the phase amplitude prediction network completed by the adversarial generation training, and the audio phase spectrum and amplitude spectrum are obtained. These spectrums are used as references to generate the audio signal generated by the adversarial generation training.
By fully considering phase information, the quality and clarity of audio signal generation are improved, the generation of artifacts is reduced, and the user experience of subsequent speech synthesis is improved.
Smart Images
Figure CN119580694A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of R&D design and audio processing technology, and is applied to the pre-processing scenario of audio synthesis, and in particular to an audio signal generation method, device, equipment and storage medium thereof. Background Art
[0002] In recent years, the existing neural vocoder technology has made significant progress in the fields of speech synthesis and sound conversion. However, the current technology still has corresponding shortcomings.
[0003] For example, many vocoders are prone to artifacts when generating audio signals. Artifacts refer to the appearance of abnormal noise points. These artifacts will seriously affect the naturalness and clarity of the audio, causing the synthesized speech to sound less realistic. Especially when processing high-frequency content, the artifact phenomenon is more obvious, affecting the user experience; at the same time, traditional vocoders usually focus on the generation of amplitude spectra and ignore phase information. In fact, phase information is crucial to the quality of audio signals. The lack of accurate modeling of the phase may lead to a decrease in the quality and clarity of the generated audio, affecting subsequent speech synthesis and user experience. Summary of the invention
[0004] The purpose of the embodiments of the present application is to propose an audio signal generation method, apparatus, device and storage medium thereof, so as to solve the problem that when generating audio signals in the prior art, there are high artifacts and phase information is ignored, resulting in a decrease in the quality and clarity of the generated audio, affecting subsequent speech synthesis and user experience.
[0005] In order to solve the above technical problems, the embodiment of the present application provides an audio signal generation method, which adopts the following technical solution:
[0006] A method for generating an audio signal comprises the following steps:
[0007] Get the target audio segment;
[0008] Inputting the target audio segment into a pre-trained audio feature extraction network to extract audio features and obtain audio features output by the audio feature extraction network;
[0009] Inputting the audio features into a phase amplitude prediction network that has completed adversarial generation training, and obtaining an audio phase spectrum and an audio amplitude spectrum output by the phase amplitude prediction network;
[0010] The audio phase spectrum and the audio amplitude spectrum are used as reference spectra for audio signal generation, and an audio signal generation network trained through adversarial generation is used to generate an audio signal for the audio features.
[0011] Furthermore, before executing the step of inputting the target audio segment into a pre-trained audio feature extraction network to extract audio features and obtain audio features output by the audio feature extraction network, the method further includes:
[0012] Acquire training samples, wherein the training samples include a batch of audio clips with marked frequency domain features, and the frequency domain features include phase information and amplitude information;
[0013] The training samples are input into the audio feature extraction network to be trained, and the audio feature extraction network is pre-trained to obtain a pre-trained audio feature extraction network.
[0014] Furthermore, the step of inputting the training sample into the audio feature extraction network to be trained and pre-training the audio feature extraction network to obtain a pre-trained audio feature extraction network specifically includes:
[0015] Step 401, using the audio feature extraction network to be trained to extract audio features from the batch of audio clips to obtain audio feature extraction results;
[0016] Step 402, obtaining the frequency domain features corresponding to each audio segment in the batch of audio segments from the audio feature extraction result as actual extracted frequency domain features;
[0017] Step 403, performing consistency comparison between the actually extracted frequency domain features and the marked frequency domain features in the training sample according to the difference in audio clips;
[0018] Step 404, according to the consistency comparison result, the number of audio clips with consistent phase information and amplitude information is counted as the frequency domain feature consistency number;
[0019] Step 405, calculating the frequency domain feature extraction accuracy of the audio feature extraction network using a proportional algorithm according to the frequency domain feature consistency number and the total number of audio segments in the training sample;
[0020] Step 406: If the frequency domain feature extraction accuracy does not exceed the preset accuracy threshold, a back propagation algorithm is used to reversely optimize and adjust the network parameters of the audio feature extraction network, and steps 401 to 405 are iteratively performed until the frequency domain feature extraction accuracy exceeds the preset accuracy threshold;
[0021] Step 407: If the frequency domain feature extraction accuracy exceeds a preset accuracy threshold, the audio feature extraction network pre-training is completed.
[0022] Furthermore, before executing the step of inputting the audio feature into the phase amplitude prediction network that has completed adversarial generation training to obtain the audio phase spectrum and audio amplitude spectrum output by the phase amplitude prediction network, the method further includes:
[0023] Obtaining audio features corresponding to all audio clips extracted from the training samples by the pre-trained audio feature extraction network;
[0024] Inputting the audio features corresponding to all the audio clips into an audio generator composed of a phase amplitude prediction network and an audio signal generation network;
[0025] The audio generator is trained for audio generation by using an adversarial generation training method to obtain a phase amplitude prediction network completed by adversarial generation training and an audio signal generation network completed by adversarial generation training.
[0026] Furthermore, the step of performing audio generation training on the audio generator by using an adversarial generation training method to obtain a phase amplitude prediction network completed by adversarial generation training and an audio signal generation network completed by adversarial generation training specifically includes:
[0027] Step 601, obtaining phase spectra and amplitude spectra respectively predicted by the phase amplitude prediction network in the audio generator based on the audio features respectively corresponding to all the audio clips;
[0028] Step 602: using the phase spectra and the amplitude spectra corresponding to all the audio clips as audio signal generation reference spectra of an audio signal generation network to generate generated audio signals corresponding to all the audio clips;
[0029] Step 603, comparing the generated audio signals corresponding to all the audio clips respectively with the real audio signals corresponding to all the audio clips respectively;
[0030] Step 604, calculating the adversarial loss value of the audio generator and the artifact loss value between the generated audio signal and the real audio signal according to the comparison result;
[0031] Step 605: If both the adversarial loss value and the artifact loss value meet the corresponding loss thresholds, the adversarial training of the audio generator is completed; otherwise, the phase amplitude prediction network and the audio signal generation network are parameter tuned and updated through the back propagation algorithm, and steps 601 to 604 are iteratively executed until both the adversarial loss value and the artifact loss value meet the corresponding loss thresholds.
[0032] Furthermore, the audio features corresponding to all the audio clips respectively include time domain features, the time domain features include duration information of the audio clips, and the step of calculating the artifact loss value between the generated audio signal and the real audio signal specifically includes:
[0033] Using a wavelet transform method or a Wiener filtering method, the abnormal noise points of the target generated audio signal compared with the corresponding real audio signal are identified;
[0034] Counting the number of abnormal noise points;
[0035] The artifact loss value is determined according to the number of the abnormal noise points and the duration of the corresponding audio segment.
[0036] Furthermore, the step of determining the artifact loss value according to the number of abnormal noise points and the duration of the corresponding audio segment specifically includes:
[0037] According to the preset algorithm formula:
[0038]
[0039] Calculating the average number of abnormal noise points of the generated audio signal within a unit time length, wherein n represents the average number of abnormal noise points, N represents the number of abnormal noise points, M represents the time length of the corresponding audio segment, and m represents a predetermined unit time length;
[0040] According to a preset artifact loss value determination form, the artifact loss value corresponding to the average number of abnormal noise points is identified, wherein the artifact loss value determination form specifies the average number of abnormal noise points intervals corresponding to different artifact loss values, and the artifact loss value and the average number of abnormal noise points interval follow the rule that the larger the artifact loss value, the larger the minimum value of the corresponding average number of abnormal noise points interval.
[0041] In order to solve the above technical problems, the embodiment of the present application also provides an audio signal generating device, which adopts the following technical solution:
[0042] An audio signal generating device, comprising:
[0043] A target audio segment acquisition module is used to acquire a target audio segment;
[0044] An audio feature extraction module, used to input the target audio segment into a pre-trained audio feature extraction network, perform audio feature extraction, and obtain audio features output by the audio feature extraction network;
[0045] A phase amplitude prediction module, used to input the audio features into a phase amplitude prediction network that has completed adversarial generation training, and obtain an audio phase spectrum and an audio amplitude spectrum output by the phase amplitude prediction network;
[0046] The audio signal generation module is used to use the audio phase spectrum and the audio amplitude spectrum as reference spectra for audio signal generation, and use an audio signal generation network trained by adversarial generation to generate an audio signal for the audio features.
[0047] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:
[0048] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned audio signal generation method when executing the computer-readable instructions.
[0049] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:
[0050] A computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the audio signal generating method as described above are implemented.
[0051] Compared with the prior art, the embodiments of the present application have the following beneficial effects:
[0052] The audio signal generation method described in the embodiment of the present application obtains a target audio segment; inputs the target audio segment into a pre-trained audio feature extraction network to extract audio features; inputs the audio features into a phase amplitude prediction network trained in adversarial generation to obtain an audio phase spectrum and an audio amplitude spectrum; uses the audio phase spectrum and the audio amplitude spectrum as reference spectra for audio signal generation, and uses the audio signal generation network trained in adversarial generation to generate audio signals for the audio features. The audio signal generation method described in the present application introduces a phase amplitude prediction network, which fully considers the phase information when generating the audio signal in the subsequent process, so as to improve the quality and clarity of the audio signal generation. It is applied to the virtual speech synthesis scenario of intelligent voice customer service in the financial technology field, and fully combines the phase information of the reference speech to improve the quality and clarity of the synthesized speech. BRIEF DESCRIPTION OF THE DRAWINGS
[0053] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0054] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;
[0055] Figure 2 is a flowchart of an embodiment of a method for generating an audio signal according to the present application;
[0056] Figure 3 is a flowchart of a specific embodiment of performing audio feature extraction network pre-training in the audio signal generation method described in the present application;
[0057] Figure 4 yes Figure 3 A flowchart of a specific embodiment of step 302 is shown;
[0058] Figure 5 is a flowchart of a specific embodiment of performing adversarial generation training of an audio generator in the audio signal generation method described in the present application;
[0059] Figure 6 yes Figure 5 A flowchart of a specific embodiment of step 503 is shown;
[0060] Figure 7 is a structural schematic diagram of an embodiment of an audio signal generating device according to the present application;
[0061] Figure 8 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION
[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0063] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0064] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.
[0065] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.
[0066] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.
[0067] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.
[0068] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .
[0069] It should be noted that the audio signal generating method provided in the embodiment of the present application is generally executed by a server, and accordingly, the audio signal generating device is generally arranged in the server.
[0070] It should be understood that Figure 1The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0071] Continue to refer Figure 2 , shows a flow chart of an embodiment of an audio signal generation method according to the present application. The audio signal generation method comprises the following steps:
[0072] Step 201: Obtain a target audio segment.
[0073] In this embodiment, the target audio segment includes a voice package of a target pronunciation object, or an audio segment recorded by the target pronunciation object for a specified text content.
[0074] By acquiring the target audio segment and using the audio signal generation method described in the present application to generate an audio signal, when realizing subsequent speech synthesis, the audio waveform is directly used as the acoustic feature of the target pronunciation object, thereby synthesizing a virtual speech based on the speech of the target pronunciation object.
[0075] Therefore, the audio signal generation method described in the present application can be applied to the pre-processing scenario of virtual voice synthesis for intelligent voice customer service in the field of financial technology, so as to generate an audio signal for the collected real reference voice before the intelligent voice customer service performs virtual voice synthesis, so as to facilitate the subsequent virtual voice synthesis.
[0076] Step 202: input the target audio segment into a pre-trained audio feature extraction network to extract audio features and obtain audio features output by the audio feature extraction network.
[0077] In this embodiment, the audio feature extraction includes extracting time domain features and frequency domain features from the target audio segment. The time domain features include duration information of the target audio segment, and the frequency domain features include amplitude information and phase information of all audio signals in the target audio segment.
[0078] By using a pre-trained audio feature extraction network, audio features are extracted from the target audio segment, so that the audio features can be used to generate an audio signal later.
[0079] Step 203: input the audio features into a phase-amplitude prediction network that has completed adversarial generation training, and obtain an audio phase spectrum and an audio amplitude spectrum output by the phase-amplitude prediction network.
[0080] The phase amplitude prediction network completed by the adversarial generation training is used to predict the audio phase spectrum and the audio amplitude spectrum of the target audio segment, so that the audio phase spectrum and the audio amplitude spectrum can be used as reference spectra for audio signal generation to generate audio signals.
[0081] Step 204: Use the audio phase spectrum and the audio amplitude spectrum as reference spectra for audio signal generation, and use an audio signal generation network trained through adversarial generation to generate an audio signal for the audio features.
[0082] By adopting an audio signal generation network trained through adversarial generation to generate audio signals from the audio features, it is ensured that a high-quality audio signal is finally generated.
[0083] In this embodiment, the target audio clip is obtained; the target audio clip is input into a pre-trained audio feature extraction network to extract audio features; the audio features are input into a phase amplitude prediction network trained in adversarial generation to obtain an audio phase spectrum and an audio amplitude spectrum; the audio phase spectrum and the audio amplitude spectrum are used as reference spectra for audio signal generation, and an audio signal generation network trained in adversarial generation is used to generate audio signals for audio features. The audio signal generation method described in the present application introduces a phase amplitude prediction network, which fully considers phase information when subsequently generating audio signals to improve the quality and clarity of audio signal generation. It is applied to the virtual speech synthesis scenario of intelligent voice customer service in the financial technology field, and fully combines the phase information of the reference speech to improve the quality and clarity of the synthesized speech.
[0084] Continue to refer Figure 3 In some optional implementations, a step of pre-training an audio feature extraction network is also included before step 202. Figure 3 The flowchart of a specific embodiment of performing audio feature extraction network pre-training in the audio signal generation method described in the present application includes the following steps:
[0085] Step 301, obtaining a training sample, wherein the training sample includes a batch of audio clips with marked frequency domain features, and the frequency domain features include phase information and amplitude information;
[0086] Step 302: input the training sample into the audio feature extraction network to be trained, and pre-train the audio feature extraction network to obtain a pre-trained audio feature extraction network.
[0087] The audio feature extraction network is pre-trained by using batches of audio clips with labeled frequency domain features to ensure that accurate phase information and amplitude information are extracted from the target audio clips during the subsequent actual audio feature extraction, thereby achieving the subsequent application of accurate phase information to audio signal generation, thereby improving the quality and clarity of audio signal generation.
[0088] Continue to refer Figure 4 , Figure 4 yes Figure 3 The flowchart of a specific embodiment of step 302 shown includes the following steps:
[0089] Step 401, using the audio feature extraction network to be trained to extract audio features from the batch of audio clips to obtain audio feature extraction results;
[0090] Step 402, obtaining the frequency domain features corresponding to each audio segment in the batch of audio segments from the audio feature extraction result as actual extracted frequency domain features;
[0091] Step 403, performing consistency comparison between the actually extracted frequency domain features and the marked frequency domain features in the training sample according to the difference in audio clips;
[0092] Specifically, the actually extracted frequency domain features corresponding to the same audio clip and the marked frequency domain features corresponding to the same audio clip in the training sample are obtained, and the phase information and amplitude information in the frequency domain features are compared to see whether the phase information and the amplitude information are consistent.
[0093] Step 404, according to the consistency comparison result, the number of audio clips with consistent phase information and amplitude information is counted as the frequency domain feature consistency number;
[0094] Step 405, calculating the frequency domain feature extraction accuracy of the audio feature extraction network using a proportional algorithm according to the frequency domain feature consistency number and the total number of audio segments in the training sample;
[0095] Specifically, the proportional algorithm uses the frequency domain feature consistency number as the numerator and the total number of audio clips in the training sample as the denominator to calculate the frequency domain feature extraction accuracy of the audio feature extraction network.
[0096] Step 406: If the frequency domain feature extraction accuracy does not exceed the preset accuracy threshold, a back propagation algorithm is used to reversely optimize and adjust the network parameters of the audio feature extraction network, and steps 401 to 405 are iteratively performed until the frequency domain feature extraction accuracy exceeds the preset accuracy threshold;
[0097] Step 407: If the frequency domain feature extraction accuracy exceeds a preset accuracy threshold, the audio feature extraction network pre-training is completed.
[0098] By pre-training the audio feature extraction network using the phase information and amplitude information in the frequency domain features, it is ensured that accurate phase information and amplitude information are extracted from the target audio segment when the audio feature extraction is actually performed later, thereby realizing the subsequent application of accurate phase information to audio signal generation, thereby improving the quality and clarity of audio signal generation.
[0099] Continue to refer Figure 5 In some optional implementations, before step 203, a step of performing adversarial generation training of an audio generator is also included. Figure 5 is a flowchart of a specific embodiment of performing adversarial generation training of an audio generator in the audio signal generation method described in the present application, comprising the following steps:
[0100] Step 501, obtaining audio features corresponding to all audio clips extracted from the training samples by the pre-trained audio feature extraction network;
[0101] Step 502, inputting the audio features corresponding to all the audio clips into an audio generator composed of a phase amplitude prediction network and an audio signal generation network;
[0102] Step 503: Perform audio generation training on the audio generator using an adversarial generation training method to obtain a phase amplitude prediction network completed by adversarial generation training and an audio signal generation network completed by adversarial generation training.
[0103] By directly using the pre-trained audio feature extraction network to extract the audio features corresponding to all the audio clips from the training samples, adversarial generation training of the audio generator is performed. Since the audio generator in the present application is composed of a phase amplitude prediction network and an audio signal generation network, adversarial generation training is no longer performed on the feature extraction network. Only adversarial generation training is performed on the phase amplitude prediction network and the audio signal generation network, thereby ensuring the high availability of the phase amplitude prediction network and the audio signal generation network when the training is completed.
[0104] Continue to refer Figure 6 , Figure 6 yes Figure 5 The flowchart of a specific embodiment of step 503 shown includes the following steps:
[0105] Step 601, obtaining phase spectra and amplitude spectra respectively predicted by the phase amplitude prediction network in the audio generator based on the audio features respectively corresponding to all the audio clips;
[0106] Step 602: using the phase spectra and the amplitude spectra corresponding to all the audio clips as audio signal generation reference spectra of an audio signal generation network to generate generated audio signals corresponding to all the audio clips;
[0107] Step 603, comparing the generated audio signals corresponding to all the audio clips respectively with the real audio signals corresponding to all the audio clips respectively;
[0108] Step 604, calculating the adversarial loss value of the audio generator and the artifact loss value between the generated audio signal and the real audio signal according to the comparison result;
[0109] Specifically, it should be understood that the adversarial loss value is the total loss value of the generated audio signal and the real audio signal for all audio clips in the entire training sample as a whole, and the artifact loss value is the signal loss value between the generated audio signal and the real audio signal in each audio clip dimension.
[0110] Step 605: If both the adversarial loss value and the artifact loss value meet the corresponding loss thresholds, the adversarial training of the audio generator is completed; otherwise, the phase amplitude prediction network and the audio signal generation network are parameter tuned and updated through the back propagation algorithm, and steps 601 to 604 are iteratively executed until both the adversarial loss value and the artifact loss value meet the corresponding loss thresholds.
[0111] By calculating the adversarial loss value and the artifact loss value, the audio generator is trained for adversarial generation, which can not only ensure the overall generation quality of the audio generator completed by the adversarial generation training, but also ensure that the audio generator controls the generation quality based on the generation details of each audio segment, and ensure the accuracy of the audio amplitude spectrum and audio phase spectrum predicted by the phase amplitude prediction network finally completed by the adversarial generation training, but also ensure the authenticity of the audio signal generated by the audio signal generation network completed by the adversarial generation training, avoid the presence of a large number of artifacts in the final generated audio signal, make the generated audio signal more natural and clear, and improve the quality of the audio signal.
[0112] In this embodiment, the audio features corresponding to all the audio clips respectively include time domain features, and the time domain features include duration information of the audio clips.
[0113] In this embodiment, the step of calculating the artifact loss value between the generated audio signal and the real audio signal specifically includes: using a wavelet transformation method or a Wiener filtering method to identify abnormal noise points of the target generated audio signal compared to the corresponding real audio signal; counting the number of the abnormal noise points; and determining the artifact loss value based on the number of the abnormal noise points and the duration of the corresponding audio segment.
[0114] Specifically, the step of determining the artifact loss value according to the number of abnormal noise points and the duration of the corresponding audio segment includes:
[0115] According to the preset algorithm formula:
[0116]
[0117] Calculating the average number of abnormal noise points of the generated audio signal within a unit time length, wherein n represents the average number of abnormal noise points, N represents the number of abnormal noise points, M represents the time length of the corresponding audio segment, and m represents a predetermined unit time length;
[0118] According to a preset artifact loss value determination form, the artifact loss value corresponding to the average number of abnormal noise points is identified, wherein the artifact loss value determination form specifies the average number of abnormal noise points intervals corresponding to different artifact loss values, and the artifact loss value and the average number of abnormal noise points interval follow the rule that the larger the artifact loss value, the larger the minimum value of the corresponding average number of abnormal noise points interval.
[0119] It should be understood that, since there is uncertainty in the abnormal noise points that may be introduced when an audio signal is generated, accordingly, if the audio signal duration is longer, the fewer abnormal noise points introduced therein, the higher the quality of the generated audio signal. Therefore, the audio signal generation method described in the present application introduces the calculation of the average number of abnormal noise points of the generated audio signal per unit duration, and measures the quality of the generated audio signal by the average number of abnormal noise points, thereby ensuring the high availability of the audio generator that has been trained to achieve adversarial generation, so that when the audio signal is actually generated, the introduction of abnormal noise points is reduced as much as possible, thereby improving the quality of the generated audio signal.
[0120] This application obtains the target audio clip; inputs the target audio clip into the pre-trained audio feature extraction network to extract audio features; inputs the audio features into the phase amplitude prediction network trained by adversarial generation to obtain the audio phase spectrum and audio amplitude spectrum; uses the audio phase spectrum and the audio amplitude spectrum as reference spectra for audio signal generation, and uses the audio signal generation network trained by adversarial generation to generate audio signals for the audio features. The audio signal generation method described in this application introduces a phase amplitude prediction network, which fully considers the phase information when generating audio signals in the subsequent process, so as to improve the quality and clarity of the audio signal generation; at the same time, in the adversarial generation training stage of the audio generator, the artifact loss calculation is introduced to ensure that the number of abnormal noise points generated is reduced when the audio signal is actually generated in the subsequent process, so as to ensure the generation of high-quality audio signals. It is applied to the virtual speech synthesis scenario of intelligent voice customer service in the financial technology field, and fully combines the phase information of the reference speech to improve the quality and clarity of the synthesized speech.
[0121] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0122] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0123] In an embodiment of the present application, a target audio segment is obtained; the target audio segment is input into a pre-trained audio feature extraction network to extract audio features; the audio features are input into a phase amplitude prediction network trained by adversarial generation to obtain an audio phase spectrum and an audio amplitude spectrum; the audio phase spectrum and the audio amplitude spectrum are used as reference spectra for audio signal generation, and an audio signal generation network trained by adversarial generation is used to generate audio signals for audio features. The audio signal generation method described in the present application introduces a phase amplitude prediction network, which fully considers phase information when performing subsequent audio signal generation to improve the quality and clarity of audio signal generation; at the same time, in the adversarial generation training stage of the audio generator, an artifact loss calculation is introduced to ensure that the number of abnormal noise points generated is reduced when the audio signal is actually generated in the subsequent generation, and to ensure the generation of high-quality audio signals. Applied to the virtual speech synthesis scenario of intelligent voice customer service in the financial technology field, the phase information of the reference speech is fully combined to improve the quality and clarity of the synthesized speech.
[0124] Further references Figure 7 , as a response to the above Figure 2 In order to realize the method shown in the figure, the present application provides an embodiment of an audio signal generating device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0125] like Figure 7 As shown, the audio signal generating device 700 described in this embodiment includes: a target audio segment acquiring module 701, an audio feature extracting module 702, a phase amplitude predicting module 703 and an audio signal generating module 704. Among them:
[0126] A target audio segment acquisition module 701 is used to acquire a target audio segment;
[0127] An audio feature extraction module 702 is used to input the target audio segment into a pre-trained audio feature extraction network to extract audio features and obtain audio features output by the audio feature extraction network;
[0128] The phase amplitude prediction module 703 is used to input the audio features into the phase amplitude prediction network that has completed adversarial generation training, and obtain the audio phase spectrum and audio amplitude spectrum output by the phase amplitude prediction network;
[0129] The audio signal generating module 704 is used to use the audio phase spectrum and the audio amplitude spectrum as reference spectra for generating an audio signal, and to generate an audio signal for the audio features using an audio signal generating network trained by adversarial generation.
[0130] This application obtains the target audio clip; inputs the target audio clip into the pre-trained audio feature extraction network to extract audio features; inputs the audio features into the phase amplitude prediction network trained by adversarial generation to obtain the audio phase spectrum and audio amplitude spectrum; uses the audio phase spectrum and the audio amplitude spectrum as reference spectra for audio signal generation, and uses the audio signal generation network trained by adversarial generation to generate audio signals for the audio features. The audio signal generation method described in this application introduces a phase amplitude prediction network, which fully considers the phase information when generating audio signals in the subsequent process, so as to improve the quality and clarity of the audio signal generation; at the same time, in the adversarial generation training stage of the audio generator, the artifact loss calculation is introduced to ensure that the number of abnormal noise points generated is reduced when the audio signal is actually generated in the subsequent process, so as to ensure the generation of high-quality audio signals. It is applied to the virtual speech synthesis scenario of intelligent voice customer service in the financial technology field, and fully combines the phase information of the reference speech to improve the quality and clarity of the synthesized speech.
[0131] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0132] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0133] To solve the above technical problems, the present application also provides a computer device. Figure 8 , Figure 8 This is a basic structural block diagram of the computer device in this embodiment.
[0134] The computer device 8 includes a memory 8a, a processor 8b, and a network interface 8c which are interconnected through a system bus. It should be noted that: Figure 8 Only a computer device 8 having a component memory 8a, a processor 8b, and a network interface 8c is shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.
[0135] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.
[0136] The memory 8a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 8a can be an internal storage unit of the computer device 8, such as a hard disk or memory of the computer device 8. In other embodiments, the memory 8a can also be an external storage device of the computer device 8, such as a plug-in hard disk equipped on the computer device 8, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. Of course, the memory 8a can also include both the internal storage unit of the computer device 8 and its external storage device. In this embodiment, the memory 8a is generally used to store the operating system and various application software installed on the computer device 8, such as a computer-readable instruction of an audio signal generation method, etc. In addition, the memory 8a can also be used to temporarily store various types of data that have been output or are to be output.
[0137] The processor 8b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 8b is generally used to control the overall operation of the computer device 8. In this embodiment, the processor 8b is used to run computer-readable instructions or process data stored in the memory 8a, such as computer-readable instructions for running the audio signal generation method.
[0138] The network interface 8c may include a wireless network interface or a wired network interface. The network interface 8c is generally used to establish a communication connection between the computer device 8 and other electronic devices.
[0139] The computer device proposed in this embodiment belongs to the field of research and development design and audio processing technology, and is applied to the pre-processing scenario of audio synthesis. This application obtains the target audio segment; inputs the target audio segment into the pre-trained audio feature extraction network to extract audio features; inputs the audio features into the phase amplitude prediction network trained by adversarial generation to obtain the audio phase spectrum and the audio amplitude spectrum; uses the audio phase spectrum and the audio amplitude spectrum as the reference spectrum for audio signal generation, and uses the audio signal generation network trained by adversarial generation to generate audio signals for audio features. The audio signal generation method described in this application introduces a phase amplitude prediction network, which fully considers the phase information when the audio signal is generated in the subsequent process, so as to improve the quality and clarity of the audio signal generation; at the same time, in the adversarial generation training stage of the audio generator, the artifact loss calculation is introduced to ensure that the number of abnormal noise points generated is reduced when the audio signal is actually generated in the subsequent process, so as to ensure the generation of high-quality audio signals. It is applied to the virtual speech synthesis scenario of intelligent voice customer service in the field of financial technology, and fully combines the phase information of the reference speech to improve the quality and clarity of the synthesized speech.
[0140] The present application also provides another implementation, that is, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the audio signal generation method as described above.
[0141] The computer-readable storage medium proposed in this embodiment belongs to the field of research and development design and audio processing technology, and is applied to the pre-processing scenario of audio synthesis. This application obtains the target audio segment; inputs the target audio segment into the pre-trained audio feature extraction network to extract audio features; inputs the audio features into the phase amplitude prediction network trained by adversarial generation to obtain the audio phase spectrum and the audio amplitude spectrum; uses the audio phase spectrum and the audio amplitude spectrum as the reference spectrum for audio signal generation, and uses the audio signal generation network trained by adversarial generation to generate audio signals for audio features. The audio signal generation method described in this application introduces a phase amplitude prediction network, which fully considers the phase information when the audio signal is generated in the subsequent process, so as to improve the quality and clarity of the audio signal generation; at the same time, in the adversarial generation training stage of the audio generator, the artifact loss calculation is introduced to ensure that the number of abnormal noise points generated is reduced when the audio signal is actually generated in the subsequent process, so as to ensure the generation of high-quality audio signals. It is applied to the virtual speech synthesis scenario of intelligent voice customer service in the field of financial technology, and fully combines the phase information of the reference speech to improve the quality and clarity of the synthesized speech.
[0142] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0143] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.
Claims
1. A method for generating an audio signal, characterized in that: The steps include: Get the target audio segment; Inputting the target audio segment into a pre-trained audio feature extraction network to extract audio features and obtain audio features output by the audio feature extraction network; Inputting the audio features into a phase amplitude prediction network that has completed adversarial generation training, and obtaining an audio phase spectrum and an audio amplitude spectrum output by the phase amplitude prediction network; The audio phase spectrum and the audio amplitude spectrum are used as reference spectra for audio signal generation, and an audio signal generation network trained through adversarial generation is used to generate an audio signal for the audio features.
2. The audio signal generating method according to claim 1, characterized in that: Before executing the step of inputting the target audio segment into a pre-trained audio feature extraction network to extract audio features and obtain audio features output by the audio feature extraction network, the method further includes: Acquire training samples, wherein the training samples include a batch of audio clips with marked frequency domain features, and the frequency domain features include phase information and amplitude information; The training samples are input into the audio feature extraction network to be trained, and the audio feature extraction network is pre-trained to obtain a pre-trained audio feature extraction network.
3. The audio signal generating method according to claim 2, characterized in that: The step of inputting the training sample into the audio feature extraction network to be trained and pre-training the audio feature extraction network to obtain a pre-trained audio feature extraction network specifically includes: Step 401, using the audio feature extraction network to be trained to extract audio features from the batch of audio clips to obtain audio feature extraction results; Step 402, obtaining the frequency domain features corresponding to each audio segment in the batch of audio segments from the audio feature extraction result as actual extracted frequency domain features; Step 403, performing consistency comparison between the actually extracted frequency domain features and the marked frequency domain features in the training sample according to the difference in audio clips; Step 404, according to the consistency comparison result, the number of audio clips with consistent phase information and amplitude information is counted as the frequency domain feature consistency number; Step 405, calculating the frequency domain feature extraction accuracy of the audio feature extraction network using a proportional algorithm according to the frequency domain feature consistency number and the total number of audio segments in the training sample; Step 406: If the frequency domain feature extraction accuracy does not exceed the preset accuracy threshold, a back propagation algorithm is used to reversely optimize and adjust the network parameters of the audio feature extraction network, and steps 401 to 405 are iteratively performed until the frequency domain feature extraction accuracy exceeds the preset accuracy threshold; Step 407: If the frequency domain feature extraction accuracy exceeds a preset accuracy threshold, the audio feature extraction network pre-training is completed.
4. The audio signal generating method according to claim 2, characterized in that: Before executing the step of inputting the audio feature into the phase amplitude prediction network that has completed adversarial generation training, and obtaining the audio phase spectrum and audio amplitude spectrum output by the phase amplitude prediction network, the method further includes: Obtaining audio features corresponding to all audio clips extracted from the training samples by the pre-trained audio feature extraction network; Inputting the audio features corresponding to all the audio clips into an audio generator composed of a phase amplitude prediction network and an audio signal generation network; The audio generator is trained for audio generation by using an adversarial generation training method to obtain a phase amplitude prediction network completed by adversarial generation training and an audio signal generation network completed by adversarial generation training.
5. The audio signal generating method according to claim 4, characterized in that: The step of performing audio generation training on the audio generator by using an adversarial generation training method to obtain a phase amplitude prediction network completed by adversarial generation training and an audio signal generation network completed by adversarial generation training specifically includes: Step 601, obtaining phase spectra and amplitude spectra respectively predicted by the phase amplitude prediction network in the audio generator based on the audio features respectively corresponding to all the audio clips; Step 602: using the phase spectra and the amplitude spectra corresponding to all the audio clips as audio signal generation reference spectra of an audio signal generation network to generate generated audio signals corresponding to all the audio clips; Step 603, comparing the generated audio signals corresponding to all the audio clips with the real audio signals corresponding to all the audio clips; Step 604, calculating the adversarial loss value of the audio generator and the artifact loss value between the generated audio signal and the real audio signal according to the comparison result; Step 605: If both the adversarial loss value and the artifact loss value meet the corresponding loss thresholds, the adversarial training of the audio generator is completed; otherwise, the phase amplitude prediction network and the audio signal generation network are parameter tuned and updated through the back propagation algorithm, and steps 601 to 604 are iteratively executed until both the adversarial loss value and the artifact loss value meet the corresponding loss thresholds.
6. The audio signal generating method according to claim 5, characterized in that: The audio features corresponding to all the audio clips include time domain features, and the time domain features include duration information of the audio clips. The step of calculating the artifact loss value between the generated audio signal and the real audio signal specifically includes: Using a wavelet transform method or a Wiener filtering method, the abnormal noise points of the target generated audio signal compared with the corresponding real audio signal are identified; Counting the number of abnormal noise points; The artifact loss value is determined according to the number of the abnormal noise points and the duration of the corresponding audio segment.
7. The audio signal generating method according to claim 6, characterized in that: The step of determining the artifact loss value according to the number of abnormal noise points and the duration of the corresponding audio segment specifically includes: According to the preset algorithm formula: Calculating the average number of abnormal noise points of the generated audio signal within a unit time length, wherein n represents the average number of abnormal noise points, N represents the number of abnormal noise points, M represents the time length of the corresponding audio segment, and m represents a predetermined unit time length; According to a preset artifact loss value determination form, the artifact loss value corresponding to the average number of abnormal noise points is identified, wherein the artifact loss value determination form specifies the average number of abnormal noise points intervals corresponding to different artifact loss values, and the artifact loss value and the average number of abnormal noise points interval follow the rule that the larger the artifact loss value, the larger the minimum value of the corresponding average number of abnormal noise points interval.
8. An audio signal generating device, characterized in that: include: A target audio segment acquisition module is used to acquire a target audio segment; An audio feature extraction module, used to input the target audio segment into a pre-trained audio feature extraction network, perform audio feature extraction, and obtain audio features output by the audio feature extraction network; A phase amplitude prediction module, used to input the audio features into a phase amplitude prediction network that has completed adversarial generation training, and obtain an audio phase spectrum and an audio amplitude spectrum output by the phase amplitude prediction network; The audio signal generation module is used to use the audio phase spectrum and the audio amplitude spectrum as reference spectra for audio signal generation, and use an audio signal generation network trained by adversarial generation to generate an audio signal for the audio features.
9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the audio signal generating method according to any one of claims 1 to 7 when executing the computer-readable instructions.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the audio signal generating method according to any one of claims 1 to 7 are implemented.