Audio waveform generation method, device and equipment and storage medium thereof
By adopting the audio feature extraction and generation network of the bidirectional neural network architecture in the neural vocoder, the problem that the one-way architecture cannot capture the bidirectional dependence of the input features is solved, and the audio generation quality is improved.
Patent Information
- Application Number
- CN202411758353.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-02
- Publication Date
- 2025-03-18
AI Technical Summary
The existing neurovocoder adopts a one-way architecture and cannot effectively capture the bidirectional dependence of input features, resulting in defects in quality details and naturalness of the output audio.
The audio feature extraction network and audio waveform generation network are adopted with a bidirectional neural network architecture. Through forward extraction and reverse extraction of audio features, time serialization marking and sorting is combined with input timing information to generate high-quality audio waveforms.
It effectively improves the quality of audio generation, enhances details and nature, and solves the problem that one-way architecture cannot capture bidirectional dependencies.
Smart Images
Figure CN119649846A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of R&D design and audio processing technology, and is applied to the pre-processing scenario of audio synthesis, and in particular to an audio waveform generation method, device, equipment and storage medium thereof. Background Art
[0002] In recent years, the existing neural vocoder technology has made significant progress in the fields of speech synthesis and sound conversion. However, the current technology still has corresponding shortcomings.
[0003] Currently, most neural vocoders use a unidirectional architecture. For example, autoencoders or unidirectional generative models cannot effectively capture the bidirectional dependencies of input features in some complex audio generation tasks, thus limiting the improvement of generation quality. Especially in multi-dimensional audio generation, such as emotional speech conversion or high-quality speech synthesis, it is easy to cause the output audio to still have certain defects in quality details and naturalness. Summary of the invention
[0004] The purpose of the embodiments of the present application is to propose an audio waveform generation method, apparatus, device and storage medium thereof, so as to solve the problem that the existing neural vocoder adopts a unidirectional architecture and cannot effectively capture the bidirectional dependency of input features, which easily leads to certain defects in the quality details and naturalness of the output audio.
[0005] In order to solve the above technical problems, the embodiment of the present application provides an audio waveform generation method, which adopts the following technical solution:
[0006] A method for generating an audio waveform comprises the following steps:
[0007] Get the target audio segment;
[0008] Inputting the target audio segment into a pre-trained audio feature extraction network to extract audio features;
[0009] Obtaining target audio features output by the audio feature extraction network;
[0010] Inputting the target audio features into a pre-trained audio waveform generation network to generate an audio waveform;
[0011] Obtain a target audio waveform output by the audio waveform generation network.
[0012] Furthermore, the audio feature extraction network is a bidirectional neural network architecture, and the bidirectional neural networks are respectively an audio feature forward extraction neural network and an audio feature reverse extraction neural network. The step of inputting the target audio segment into the pre-trained audio feature extraction network to extract the audio features specifically includes:
[0013] Using the audio feature forward extraction neural network, forward extracting audio features of the target audio segment along the input direction of the target audio segment to obtain a first audio feature extraction result;
[0014] Using the audio feature reverse extraction neural network, reversely extract the audio features of the target audio segment along a direction opposite to the input direction of the target audio segment to obtain a second audio feature extraction result;
[0015] The step of obtaining the target audio features output by the audio feature extraction network specifically includes:
[0016] The first audio feature extraction result and the second audio feature extraction result are output together as target audio features.
[0017] Furthermore, before executing the step of inputting the target audio feature into the pre-trained audio waveform generation network to generate the audio waveform, the method further includes:
[0018] Identifying input timing information corresponding to the target audio segment;
[0019] According to the input timing information, the first audio feature extraction result is marked and sorted in time series;
[0020] According to the input timing information, the second audio feature extraction result is marked and sorted in time series;
[0021] After being marked and sorted by time series, a first audio feature extraction result and a second audio feature extraction result that correspond to each other in time sequence are obtained.
[0022] Furthermore, the audio waveform generation network is also a bidirectional neural network architecture, and the bidirectional neural networks are respectively an audio waveform forward generation neural network and an audio waveform reverse correction neural network. The step of inputting the target audio feature into the pre-trained audio waveform generation network to generate the audio waveform specifically includes:
[0023] Based on the first audio feature extraction result, using the audio waveform forward generation neural network to perform initial audio waveform generation on the target audio segment;
[0024] According to the second audio feature extraction result, using the audio waveform reverse correction neural network to reversely correct the initial audio waveform generation result;
[0025] The step of obtaining the target audio waveform output by the audio waveform generation network specifically includes:
[0026] The inversely corrected audio waveform is obtained as the target audio waveform.
[0027] Furthermore, the step of reversely correcting the initial generation result of the audio waveform using the audio waveform reverse correction neural network according to the second audio feature extraction result specifically includes:
[0028] Identify a time sequence correspondence between the second audio feature extraction result and the first audio feature extraction result;
[0029] The time sequence correspondence is used as a correction control condition of the audio waveform reverse correction neural network to reversely correct the initial generation result of the audio waveform.
[0030] Furthermore, before executing the step of inputting the target audio segment into a pre-trained audio feature extraction network to extract audio features, the method further includes:
[0031] Acquire training samples, wherein the training samples include batches of audio clips with annotated audio waveforms;
[0032] Inputting the training sample into an audio generator composed of the audio feature extraction network and the audio waveform generation network, and performing audio waveform generation training on the audio generator;
[0033] A pre-trained audio feature extraction network is obtained, and a pre-trained audio waveform generation network is obtained.
[0034] Furthermore, the step of inputting the training sample into an audio generator composed of the audio feature extraction network and the audio waveform generation network, and performing audio waveform generation training on the audio generator specifically includes:
[0035] Step 801, obtaining a first audio feature extraction result and a second audio feature extraction result extracted by the audio feature extraction network based on the training sample;
[0036] Step 802, performing distinguishing marking processing on the first audio feature extraction result and the second audio feature extraction result respectively according to the difference of the audio segments in the training sample;
[0037] Step 803, obtaining the first audio feature extraction result and the second audio feature extraction result corresponding to each audio segment in the training sample after the distinguishing mark processing;
[0038] Step 804, inputting the first audio feature extraction result and the second audio feature extraction result corresponding to each audio segment in the training sample into the audio waveform generation network in sequence according to different audio segments, and outputting the audio waveform corresponding to each audio segment in the training sample;
[0039] Step 805: input the audio waveform corresponding to each audio segment in the training sample into a preset discriminator, and compare the consistency with the labeled audio waveform corresponding to each audio segment in the training sample;
[0040] Step 806: If the consistency comparison result does not meet the preset comparison result requirement, the network parameter tuning process is performed on the audio feature extraction network and the audio waveform generation network, and steps 801 to 805 are re-executed until the consistency comparison result meets the preset comparison result requirement;
[0041] Step 807: If the consistency comparison result meets the preset comparison result requirement, the audio generator training is completed.
[0042] In order to solve the above technical problems, the embodiment of the present application also provides an audio waveform generating device, which adopts the following technical solution:
[0043] An audio waveform generating device, comprising:
[0044] A target audio segment acquisition module is used to acquire a target audio segment;
[0045] An audio feature extraction module, used to input the target audio segment into a pre-trained audio feature extraction network to extract audio features;
[0046] An audio feature output acquisition module, used to acquire the target audio features output by the audio feature extraction network;
[0047] An audio waveform generation module, used to input the target audio features into a pre-trained audio waveform generation network to generate an audio waveform;
[0048] The audio waveform output acquisition module is used to obtain the target audio waveform output by the audio waveform generation network.
[0049] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:
[0050] A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned audio waveform generation method when executing the computer-readable instructions.
[0051] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:
[0052] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the audio waveform generation method as described above.
[0053] Compared with the prior art, the embodiments of the present application have the following beneficial effects:
[0054] The audio waveform generation method described in the embodiment of the present application obtains a target audio segment; inputs the target audio segment into a pre-trained audio feature extraction network to extract audio features; obtains the target audio features output by the audio feature extraction network; inputs the target audio features into a pre-trained audio waveform generation network to generate an audio waveform; and obtains the target audio waveform output by the audio waveform generation network. The audio waveform generation method described in the present application uses a pre-trained audio feature extraction network and an audio waveform generation network to successively perform audio feature extraction and audio waveform generation, thereby ensuring the high availability of the final generated audio waveform. It is applied to the pre-processing scenario of virtual speech synthesis for intelligent voice customer service in the financial technology field, so as to perform audio waveform processing on the collected real reference speech before the intelligent voice customer service performs virtual speech synthesis, so as to facilitate the subsequent use of virtual speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS
[0055] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0056] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;
[0057] Figure 2 is a flow chart of an embodiment of an audio waveform generation method according to the present application;
[0058] Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 202 is shown;
[0059] Figure 4 It is a flowchart of a specific embodiment of marking and arranging the timing correspondence relationship in the audio waveform generation method described in the present application;
[0060] Figure 5 yes Figure 2 A flowchart of a specific embodiment of step 204 is shown;
[0061] Figure 6 yes Figure 5 A flowchart of a specific embodiment of step 502 is shown;
[0062] Figure 7 is a flowchart of a specific embodiment of pre-training an audio generator in the audio waveform generation method described in the present application;
[0063] Figure 8 yes Figure 7 A flowchart of a specific embodiment of step 702 is shown;
[0064] Fig. 9 is a structural schematic diagram of an embodiment of an audio waveform generating device according to the present application;
[0065] Fig.10 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION
[0066] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0067] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0068] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.
[0069] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.
[0070] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.
[0071] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.
[0072] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .
[0073] It should be noted that the audio waveform generation method provided in the embodiment of the present application is generally executed by a server, and accordingly, the audio waveform generation device is generally arranged in the server.
[0074] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0075] Continue to refer Figure 2 , shows a flow chart of an embodiment of an audio waveform generation method according to the present application. The audio waveform generation method comprises the following steps:
[0076] Step 201: Obtain a target audio segment.
[0077] In this embodiment, the target audio segment includes a voice package of a target pronunciation object, or an audio segment recorded by the target pronunciation object for a specified text content.
[0078] By acquiring the target audio segment and using the audio waveform generation method described in the present application to generate an audio waveform, when realizing subsequent speech synthesis, the audio waveform is directly used as the acoustic feature of the target pronunciation object, thereby synthesizing a virtual speech based on the speech of the target pronunciation object.
[0079] Therefore, the audio waveform generation method described in the present application can be applied to the pre-processing scenario of virtual voice synthesis for intelligent voice customer service in the field of financial technology, so as to perform audio waveform processing on the collected real reference voice before the intelligent voice customer service performs virtual voice synthesis, so as to facilitate the subsequent virtual voice synthesis.
[0080] Step 202: input the target audio segment into a pre-trained audio feature extraction network to extract audio features.
[0081] In this embodiment, the audio feature extraction includes extracting time domain features and frequency domain features from the target audio segment. The time domain features include time point information corresponding to all phonemes in the target audio segment, and the frequency domain features include amplitude and phase information corresponding to all audio signals in the target audio segment in the mel spectrum.
[0082] Step 203: Obtain target audio features output by the audio feature extraction network.
[0083] By using a pre-trained audio feature extraction network, audio features are extracted from the target audio segment, so that the audio features can be used to generate an audio waveform later.
[0084] Step 204: input the target audio features into a pre-trained audio waveform generation network to generate an audio waveform.
[0085] Step 205: Obtain the target audio waveform output by the audio waveform generation network.
[0086] By adopting a pre-trained audio waveform generation network, a target audio waveform is generated for the target audio segment, so that the target audio waveform can be subsequently used to perform virtual speech synthesis in an intelligent voice customer service scenario.
[0087] In this embodiment, by obtaining a target audio clip; inputting the target audio clip into a pre-trained audio feature extraction network to perform audio feature extraction; obtaining the target audio features output by the audio feature extraction network; inputting the target audio features into a pre-trained audio waveform generation network to generate an audio waveform; obtaining the target audio waveform output by the audio waveform generation network. The audio waveform generation method described in the present application, by adopting a pre-trained audio feature extraction network and an audio waveform generation network, successively performs audio feature extraction and audio waveform generation, thereby ensuring the high availability of the final generated audio waveform. It is applied to the pre-processing scenario of virtual speech synthesis for intelligent voice customer service in the financial technology field, so as to perform audio waveform processing on the collected real reference speech before the intelligent voice customer service performs virtual speech synthesis, so as to facilitate the subsequent use of virtual speech synthesis.
[0088] In this embodiment, the audio feature extraction network is a bidirectional neural network architecture, and the bidirectional neural networks are respectively an audio feature forward extraction neural network and an audio feature reverse extraction neural network. Specifically, the bidirectional neural network, for example, a bidirectional temporal neural network LSTM (Long Short Term Memory), uses a bidirectional temporal neural network LSTM, which can better capture the context feature dependency information in the target audio segment.
[0089] Continue to refer Figure 3 , Figure 3 yes Figure 2 The flowchart of a specific embodiment of step 202 shown includes the following steps:
[0090] Step 301, using the audio feature forward extraction neural network, forward extracting audio features of the target audio segment along the input direction of the target audio segment to obtain a first audio feature extraction result;
[0091] Step 302, using the audio feature reverse extraction neural network, performing audio feature reverse extraction on the target audio segment along a direction opposite to the input direction of the target audio segment, to obtain a second audio feature extraction result;
[0092] In this embodiment, the step of obtaining the target audio features output by the audio feature extraction network specifically includes:
[0093] Step 303: Output the first audio feature extraction result and the second audio feature extraction result together as target audio features.
[0094] By adopting a bidirectional neural network architecture to perform forward audio feature extraction and reverse audio feature extraction respectively, it is possible to extract audio features from both past and future contexts when performing audio feature extraction, unlike the traditional unidirectional network which only relies on unidirectional feature extraction. This bidirectional extraction method effectively improves the expressiveness of features, especially for complex audio signals, the bidirectional neural network described in this application can capture more feature information, thereby improving the quality of the subsequently generated audio waveform.
[0095] Continue to refer Figure 4 In some optional implementations, a step of marking and arranging the time sequence correspondence is also included before step 204. Figure 4 This is a flowchart of a specific embodiment of performing timing correspondence mark arrangement in the audio waveform generation method described in the present application, comprising the following steps:
[0096] Step 401, identifying input timing information corresponding to the target audio segment;
[0097] Step 402: Time-series-mark and sort the first audio feature extraction result according to the input timing information;
[0098] Step 403, marking and arranging the second audio feature extraction result in time series according to the input time sequence information;
[0099] Step 404, obtaining the first audio feature extraction result and the second audio feature extraction result that correspond to each other in time sequence after being marked and sorted by time series.
[0100] By combining the forward extraction results and reverse extraction results of audio features extracted by the bidirectional neural network with the input timing information corresponding to the target audio segment, the timing correspondence is marked and sorted to ensure timing control during subsequent audio waveform generation, thereby fully ensuring the quality of the subsequently generated audio waveform.
[0101] In this embodiment, the audio waveform generation network is also a bidirectional neural network architecture, and the bidirectional neural networks are respectively an audio waveform forward generation neural network and an audio waveform reverse correction neural network. Specifically, the bidirectional neural network is, for example, a bidirectional temporal neural network LSTM.
[0102] Continue to refer Figure 5 , Figure 5 yes Figure 2 The flowchart of a specific embodiment of step 204 shown includes the following steps:
[0103] Step 501, based on the first audio feature extraction result, using the audio waveform forward generation neural network to perform initial audio waveform generation on the target audio segment;
[0104] Step 502, according to the second audio feature extraction result, using the audio waveform reverse correction neural network to reversely correct the initial audio waveform generation result;
[0105] In this embodiment, the step of obtaining the target audio waveform output by the audio waveform generation network specifically includes:
[0106] Step 503: Obtain the reversely corrected audio waveform as the target audio waveform.
[0107] By adopting a bidirectional neural network architecture to perform initial audio waveform generation and reverse correction on the initial audio waveform generation results, since the audio feature extraction results for initial audio waveform generation and reverse correction on the initial audio waveform generation results are extracted using a bidirectional neural network, more contextual feature information in the audio clip is fully captured, so that the finally generated audio waveform has higher quality.
[0108] Continue to refer Figure 6 , Figure 6 yes Figure 5 The flowchart of a specific embodiment of step 502 shown includes the following steps:
[0109] Step 601, identifying a time sequence correspondence between a second audio feature extraction result and a first audio feature extraction result;
[0110] Step 602: Use the time sequence correspondence as a correction control condition of the audio waveform reverse correction neural network to reversely correct the initial generation result of the audio waveform.
[0111] In this embodiment, the initial generation result of the audio waveform is reversely corrected according to the time sequence correspondence between the second audio feature extraction result and the first audio feature extraction result, thereby ensuring the rationality of the reverse correction.
[0112] Continue to refer Figure 7 In some optional implementations, a step of pre-training an audio generator is also included before step 202. Figure 7 1 is a flowchart of a specific embodiment of pre-training an audio generator in the audio waveform generation method described in the present application, comprising the following steps:
[0113] Step 701, obtaining training samples, wherein the training samples include batches of audio clips with annotated audio waveforms;
[0114] Step 702, inputting the training sample into an audio generator composed of the audio feature extraction network and the audio waveform generation network, and performing audio waveform generation training on the audio generator;
[0115] Step 703, obtaining a pre-trained audio feature extraction network and a pre-trained audio waveform generation network.
[0116] By performing audio waveform generation training on the audio generator to obtain a pre-trained audio feature extraction network and a pre-trained audio waveform generation network, high availability is ensured during subsequent actual audio waveform generation.
[0117] Continue to refer Figure 8 , Figure 8 yes Figure 7 The flowchart of a specific embodiment of step 702 shown includes the following steps:
[0118] Step 801, obtaining a first audio feature extraction result and a second audio feature extraction result extracted by the audio feature extraction network based on the training sample;
[0119] Step 802, performing distinguishing marking processing on the first audio feature extraction result and the second audio feature extraction result respectively according to the difference of the audio segments in the training sample;
[0120] Step 803, obtaining the first audio feature extraction result and the second audio feature extraction result corresponding to each audio segment in the training sample after the distinguishing mark processing;
[0121] Step 804, inputting the first audio feature extraction result and the second audio feature extraction result corresponding to each audio segment in the training sample into the audio waveform generation network in sequence according to different audio segments, and outputting the audio waveform corresponding to each audio segment in the training sample;
[0122] Step 805: input the audio waveform corresponding to each audio segment in the training sample into a preset discriminator, and compare the consistency with the labeled audio waveform corresponding to each audio segment in the training sample;
[0123] Step 806: If the consistency comparison result does not meet the preset comparison result requirement, the network parameter tuning process is performed on the audio feature extraction network and the audio waveform generation network, and steps 801 to 805 are re-executed until the consistency comparison result meets the preset comparison result requirement;
[0124] Specifically, the comparison result requirements include comparison result requirements based on the entire training sample and comparison result requirements based on each audio segment, and the specific conditions are set by the trainer. The comparison result requirements based on the entire training sample, for example, are: the total number of audio segments that meet the comparison results reaches 95%; the comparison result requirements based on each audio segment, for example, are: the waveform similarity between the generated audio waveform of the audio segment used in training and the corresponding annotated audio waveform reaches a preset similarity threshold.
[0125] Step 807: If the consistency comparison result meets the preset comparison result requirement, the audio generator training is completed.
[0126] Specifically, an adversarial generative training method is adopted, and the audio generator is trained using a discriminator and combined with labeled audio waveforms in training samples, thereby ensuring the high availability of the audio feature extraction network and audio waveform generation network that are finally trained.
[0127] The present application obtains a target audio clip; inputs the target audio clip into a pre-trained audio feature extraction network to extract audio features; obtains target audio features output by the audio feature extraction network; inputs the target audio features into a pre-trained audio waveform generation network to generate an audio waveform; and obtains the target audio waveform output by the audio waveform generation network. The audio waveform generation method described in the present application uses a pre-trained audio feature extraction network and an audio waveform generation network to successively perform audio feature extraction and audio waveform generation, thereby ensuring the high availability of the final generated audio waveform. It is applied to the pre-processing scenario of virtual speech synthesis for intelligent voice customer service in the financial technology field, so as to perform audio waveform processing on the collected real reference speech before the intelligent voice customer service performs virtual speech synthesis, so as to facilitate the subsequent use of virtual speech synthesis.
[0128] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0129] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0130] In an embodiment of the present application, a target audio segment is obtained; the target audio segment is input into a pre-trained audio feature extraction network to extract audio features; the target audio features output by the audio feature extraction network are obtained; the target audio features are input into a pre-trained audio waveform generation network to generate an audio waveform; and the target audio waveform output by the audio waveform generation network is obtained. The audio waveform generation method described in the present application uses a pre-trained audio feature extraction network and an audio waveform generation network to successively perform audio feature extraction and audio waveform generation, thereby ensuring the high availability of the final generated audio waveform. It is applied to the pre-processing scenario of virtual speech synthesis for intelligent voice customer service in the financial technology field, so as to perform audio waveform processing on the collected real reference speech before the intelligent voice customer service performs virtual speech synthesis, so as to facilitate the subsequent use of virtual speech synthesis.
[0131] Further references Fig. 9 , as a response to the above Figure 2 The present application provides an embodiment of an audio waveform generating device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0132] like Fig. 9 As shown, the audio waveform generating device 900 described in this embodiment includes: a target audio segment acquiring module 901, an audio feature extracting module 902, an audio feature output acquiring module 903, an audio waveform generating module 904 and an audio waveform output acquiring module 905. Among them:
[0133] A target audio segment acquisition module 901 is used to acquire a target audio segment;
[0134] An audio feature extraction module 902 is used to input the target audio segment into a pre-trained audio feature extraction network to extract audio features;
[0135] An audio feature output acquisition module 903 is used to acquire target audio features output by the audio feature extraction network;
[0136] An audio waveform generation module 904 is used to input the target audio features into a pre-trained audio waveform generation network to generate an audio waveform;
[0137] The audio waveform output acquisition module 905 is used to obtain the target audio waveform output by the audio waveform generation network.
[0138] The present application obtains a target audio clip; inputs the target audio clip into a pre-trained audio feature extraction network to extract audio features; obtains target audio features output by the audio feature extraction network; inputs the target audio features into a pre-trained audio waveform generation network to generate an audio waveform; and obtains the target audio waveform output by the audio waveform generation network. The audio waveform generation method described in the present application uses a pre-trained audio feature extraction network and an audio waveform generation network to successively perform audio feature extraction and audio waveform generation, thereby ensuring the high availability of the final generated audio waveform. It is applied to the pre-processing scenario of virtual speech synthesis for intelligent voice customer service in the financial technology field, so as to perform audio waveform processing on the collected real reference speech before the intelligent voice customer service performs virtual speech synthesis, so as to facilitate the subsequent use of virtual speech synthesis.
[0139] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0140] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0141] To solve the above technical problems, the present application also provides a computer device. Fig.10 , Fig.10 This is a basic structural block diagram of the computer device in this embodiment.
[0142] The computer device 10 includes a memory 10a, a processor 10b, and a network interface 10c which are interconnected and communicated via a system bus. It should be noted that: Fig.10Only a computer device 10 having a component memory 10a, a processor 10b, and a network interface 10c is shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.
[0143] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.
[0144] The memory 10a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10a can be an internal storage unit of the computer device 10, such as a hard disk or memory of the computer device 10. In other embodiments, the memory 10a can also be an external storage device of the computer device 10, such as a plug-in hard disk equipped on the computer device 10, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 10a can also include both the internal storage unit of the computer device 10 and its external storage device. In this embodiment, the memory 10a is generally used to store the operating system and various application software installed on the computer device 10, such as a computer-readable instruction of an audio waveform generation method, etc. In addition, the memory 10a can also be used to temporarily store various types of data that have been output or are to be output.
[0145] The processor 10b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 10b is generally used to control the overall operation of the computer device 10. In this embodiment, the processor 10b is used to run the computer-readable instructions stored in the memory 10a or process data, such as computer-readable instructions for running the audio waveform generation method.
[0146] The network interface 10c may include a wireless network interface or a wired network interface. The network interface 10c is generally used to establish a communication connection between the computer device 10 and other electronic devices.
[0147] The computer device proposed in this embodiment belongs to the field of research and development design and audio processing technology, and is applied to the pre-processing scenario of audio synthesis. This application obtains the target audio segment; inputs the target audio segment into a pre-trained audio feature extraction network to extract audio features; obtains the target audio features output by the audio feature extraction network; inputs the target audio features into a pre-trained audio waveform generation network to generate an audio waveform; and obtains the target audio waveform output by the audio waveform generation network. The audio waveform generation method described in this application, by adopting a pre-trained audio feature extraction network and an audio waveform generation network, successively performs audio feature extraction and audio waveform generation, thereby ensuring the high availability of the final generated audio waveform. It is applied to the pre-processing scenario of virtual speech synthesis for intelligent voice customer service in the field of financial technology, so as to perform audio waveform processing on the collected real reference voice before the intelligent voice customer service performs virtual speech synthesis, so as to facilitate the subsequent use of virtual speech synthesis.
[0148] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the audio waveform generation method as described above.
[0149] The computer-readable storage medium proposed in this embodiment belongs to the field of research and development design and audio processing technology, and is applied to the pre-processing scenario of audio synthesis. This application obtains the target audio segment; inputs the target audio segment into a pre-trained audio feature extraction network to extract audio features; obtains the target audio features output by the audio feature extraction network; inputs the target audio features into a pre-trained audio waveform generation network to generate an audio waveform; and obtains the target audio waveform output by the audio waveform generation network. The audio waveform generation method described in this application, by adopting a pre-trained audio feature extraction network and an audio waveform generation network, successively performs audio feature extraction and audio waveform generation, thereby ensuring the high availability of the final generated audio waveform. It is applied to the pre-processing scenario of virtual speech synthesis for intelligent voice customer service in the field of financial technology, so as to perform audio waveform processing on the collected real reference voice before the intelligent voice customer service performs virtual speech synthesis, so as to facilitate the subsequent use of virtual speech synthesis.
[0150] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0151] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.
Claims
1. A method for generating an audio waveform, characterized in that: The steps include: Get the target audio segment; Inputting the target audio segment into a pre-trained audio feature extraction network to extract audio features; Obtaining target audio features output by the audio feature extraction network; Inputting the target audio features into a pre-trained audio waveform generation network to generate an audio waveform; Obtain a target audio waveform output by the audio waveform generation network.
2. The audio waveform generating method according to claim 1, characterized in that: The audio feature extraction network is a bidirectional neural network architecture, and the bidirectional neural networks are respectively an audio feature forward extraction neural network and an audio feature reverse extraction neural network. The step of inputting the target audio segment into the pre-trained audio feature extraction network to extract the audio features specifically includes: Using the audio feature forward extraction neural network, forward extracting audio features of the target audio segment along the input direction of the target audio segment to obtain a first audio feature extraction result; Using the audio feature reverse extraction neural network, reversely extract the audio features of the target audio segment along a direction opposite to the input direction of the target audio segment to obtain a second audio feature extraction result; The step of obtaining the target audio features output by the audio feature extraction network specifically includes: The first audio feature extraction result and the second audio feature extraction result are output together as target audio features.
3. The audio waveform generating method according to claim 2, characterized in that: Before executing the step of inputting the target audio feature into the pre-trained audio waveform generation network to generate the audio waveform, the method further includes: Identifying input timing information corresponding to the target audio segment; According to the input timing information, the first audio feature extraction result is marked and sorted in time series; According to the input timing information, the second audio feature extraction result is marked and sorted in time series; A first audio feature extraction result and a second audio feature extraction result that correspond to each other in time sequence after being marked and sorted in time series are obtained.
4. The audio waveform generating method according to claim 3, characterized in that: The audio waveform generation network is also a bidirectional neural network architecture, and the bidirectional neural networks are respectively an audio waveform forward generation neural network and an audio waveform reverse correction neural network. The step of inputting the target audio feature into the pre-trained audio waveform generation network to generate the audio waveform specifically includes: Based on the first audio feature extraction result, using the audio waveform forward generation neural network to perform initial audio waveform generation on the target audio segment; According to the second audio feature extraction result, using the audio waveform reverse correction neural network to reversely correct the initial audio waveform generation result; The step of obtaining the target audio waveform output by the audio waveform generation network specifically includes: The inversely corrected audio waveform is obtained as the target audio waveform.
5. The audio waveform generating method according to claim 4, characterized in that: The step of reversely correcting the initial generation result of the audio waveform using the audio waveform reverse correction neural network according to the second audio feature extraction result specifically includes: Identify a time sequence correspondence between the second audio feature extraction result and the first audio feature extraction result; The time sequence correspondence is used as a correction control condition of the audio waveform reverse correction neural network to reversely correct the initial generation result of the audio waveform.
6. The audio waveform generating method according to claim 1 or 2, characterized in that: Before executing the step of inputting the target audio segment into a pre-trained audio feature extraction network to extract audio features, the method further includes: Acquire training samples, wherein the training samples include batches of audio clips with labeled audio waveforms; Inputting the training sample into an audio generator composed of the audio feature extraction network and the audio waveform generation network, and performing audio waveform generation training on the audio generator; A pre-trained audio feature extraction network is obtained, and a pre-trained audio waveform generation network is obtained.
7. The audio waveform generating method according to claim 6, characterized in that: The step of inputting the training sample into an audio generator composed of the audio feature extraction network and the audio waveform generation network, and performing audio waveform generation training on the audio generator specifically includes: Step 801, obtaining a first audio feature extraction result and a second audio feature extraction result extracted by the audio feature extraction network based on the training sample; Step 802, performing distinguishing marking processing on the first audio feature extraction result and the second audio feature extraction result respectively according to the difference of the audio segments in the training sample; Step 803, obtaining the first audio feature extraction result and the second audio feature extraction result corresponding to each audio segment in the training sample after the distinguishing mark processing; Step 804, inputting the first audio feature extraction result and the second audio feature extraction result corresponding to each audio segment in the training sample into the audio waveform generation network in sequence according to different audio segments, and outputting the audio waveform corresponding to each audio segment in the training sample; Step 805: input the audio waveform corresponding to each audio segment in the training sample into a preset discriminator, and compare the consistency with the labeled audio waveform corresponding to each audio segment in the training sample; Step 806: If the consistency comparison result does not meet the preset comparison result requirement, the network parameter tuning process is performed on the audio feature extraction network and the audio waveform generation network, and steps 801 to 805 are re-executed until the consistency comparison result meets the preset comparison result requirement; Step 807: If the consistency comparison result meets the preset comparison result requirement, the audio generator training is completed.
8. An audio waveform generating device, characterized in that: include: A target audio segment acquisition module is used to acquire a target audio segment; An audio feature extraction module, used to input the target audio segment into a pre-trained audio feature extraction network to extract audio features; An audio feature output acquisition module, used to acquire the target audio features output by the audio feature extraction network; An audio waveform generation module, used to input the target audio features into a pre-trained audio waveform generation network to generate an audio waveform; The audio waveform output acquisition module is used to obtain the target audio waveform output by the audio waveform generation network.
9. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the audio waveform generation method as described in any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the audio waveform generation method according to any one of claims 1 to 7 are implemented.