Voice reconstruction method, device and equipment and storage medium thereof
By using non-upsampled convolutional layer and multi-frequency subband convolutional layer to process audio features in speech reconstruction technology, the problem of low speech reconstruction efficiency caused by high computational volume in the prior art is solved, and more efficient speech reconstruction is achieved .
Patent Information
- Application Number
- CN202411378310.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-29
- Publication Date
- 2025-01-07
AI Technical Summary
In the existing voice reconstruction technology, mainstream vocoders involve large amounts of computation due to upsampling and inverse Fourier transform, resulting in slow processing speed and reduce the efficiency of speech reconstruction.
A speech reconstruction method is adopted to obtain the original audio waveform and input it into the preset convolutional processing structure for global normalization and frequency subband decomposition processing, and use non-upsampled convolutional layer and multi-frequency subband convolutional layer. The audio features are extracted and the reconstructed speech of the target length is generated by the linear processing structure.
This reduces the amount of voice reconstruction, improves the efficiency of voice reconstruction, and improves the performance and efficiency of vocoder.
Smart Images
Figure CN119274564A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of speech processing technology, and is applied to a speech reconstruction scenario based on acoustic features, and in particular to a speech reconstruction method, apparatus, device, and storage medium thereof. Background Art
[0002] Speech reconstruction technology refers to the use of a vocoder to reconstruct or reconstruct acoustic features to obtain a speech signal. Currently, speech reconstruction technology has developed rapidly. The current mainstream vocoders are divided into the following categories: the first is a vocoder that combines audio feature upsampling and inverse Fourier transform, and the second is a vocoder based solely on inverse Fourier transform.
[0003] However, the vocoder involves upsampling and inverse Fourier transform, which involves a large amount of calculation, resulting in a slow processing speed of the vocoder and reduced speech reconstruction efficiency. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide a speech reconstruction method, apparatus, device and storage medium thereof to solve the problem that when performing existing speech reconstruction, mainstream vocoders have slow processing speeds and reduce speech reconstruction efficiency.
[0005] In order to solve the above technical problems, the present application provides a speech reconstruction method, which adopts the following technical solutions:
[0006] A speech reconstruction method comprises the following steps:
[0007] Acquiring acoustic features for speech signal reconstruction, wherein the acoustic features include original audio waveforms;
[0008] Inputting the original audio waveform into a preset first convolution processing structure to obtain a globally normalized audio feature representation output after being processed by the first convolution processing structure;
[0009] Inputting the globally normalized audio feature representation into a preset second convolution processing structure to obtain audio feature representations of multiple frequency subbands output after being processed by the second convolution processing structure;
[0010] The audio feature representations of the multiple frequency sub-bands are subjected to linear transformation processing by a preset linear processing structure to generate reconstructed speech of a target length.
[0011] Furthermore, the preset first convolution processing structure includes a first conventional convolution layer and N convolution blocks, and the step of inputting the original audio waveform into the preset first convolution processing structure to obtain a globally normalized audio feature representation output after being processed by the first convolution processing structure specifically includes:
[0012] Inputting the original audio waveform into the first conventional convolutional layer to extract audio features and obtain a first audio feature matrix;
[0013] After the first audio feature matrix is processed by N layers of convolution blocks, a globally normalized audio feature representation is obtained, where N is a positive integer.
[0014] Further, each of the convolution blocks is stacked with a deep hierarchical convolution layer, a linear normalization layer, a second conventional convolution layer, a GELU activation layer, a third conventional convolution layer and a global normalization layer in a sequential processing relationship, and the step of obtaining a globally normalized audio feature representation after processing the first audio feature matrix by N layers of convolution blocks specifically includes:
[0015] Using the deep hierarchical convolutional layer to extract low-dimensional audio features, local audio features and global audio features from the audio features included in the first audio feature matrix in sequence;
[0016] In combination with the linear normalization layer, the extracted low-dimensional audio features, local audio features and global audio features are respectively normalized to obtain normalized processing results;
[0017] According to the second conventional convolution layer and the GELU activation layer, performing conventional convolution and nonlinear introduction operations on the normalization processing result to obtain an activation representation result;
[0018] The third conventional convolution layer is used to perform a conventional convolution operation on the activation representation result again, and the convolution operation result is introduced into the global normalization layer for global normalization processing to obtain the globally normalized audio feature representation.
[0019] Further, the preset second convolution processing structure includes a fourth conventional convolution layer and a multi-frequency sub-band convolution layer, and the step of inputting the globally normalized audio feature representation into the preset second convolution processing structure to obtain the audio feature representation of multiple frequency sub-bands output after being processed by the second convolution processing structure specifically includes:
[0020] Inputting the globally normalized audio feature representation into the fourth conventional convolutional layer, and obtaining a second audio feature matrix after convolution processing by the fourth conventional convolutional layer;
[0021] The second audio feature matrix is input into the multi-frequency sub-band convolution layer, and the second audio feature matrix is decomposed by the multi-frequency sub-band convolution layer to obtain audio feature representations of multiple frequency sub-bands.
[0022] Furthermore, the multi-frequency sub-band convolution layer includes M convolution layers of different frequency sub-bands, and the step of inputting the second audio feature matrix into the multi-frequency sub-band convolution layer, decomposing the second audio feature matrix through the multi-frequency sub-band convolution layer, and obtaining audio feature representations of multiple frequency sub-bands specifically includes:
[0023] Using the M convolution layers of different frequency sub-bands to perform convolution operations on the second audio feature matrix respectively, to obtain audio feature convolution results corresponding to the M convolution layers of different frequency sub-bands respectively, where M is a positive integer;
[0024] The audio feature convolution results respectively corresponding to the convolution layers of the M different frequency sub-bands are used as audio feature representations of multiple frequency sub-bands obtained by decomposing the second audio feature matrix.
[0025] Furthermore, the preset linear processing structure includes a conventional linear layer and a reconstruction linear layer, and the step of generating a reconstructed speech of a target length after the audio feature representations of the multiple frequency subbands are subjected to linear transformation processing by the preset linear processing structure specifically includes:
[0026] Using the conventional linear layer, conventional linear transformation processing is performed on the audio feature representations of the multiple frequency sub-bands to obtain a linear transformation processing result;
[0027] The linear transformation processing result is subjected to a reconstructed linear transformation processing by using the reconstructed linear layer to generate a reconstructed speech signal of a target length, wherein the reconstruction purpose of the reconstructed linear layer is:
[0028] The audio features obtained according to the acoustic features are aligned with the target reconstructed speech to be generated.
[0029] Furthermore, all convolutional layers use a non-upsampling method when performing convolution operations, and all convolutional layers have independent convolution kernels.
[0030] In order to solve the above technical problems, the embodiment of the present application also provides a speech reconstruction device, which adopts the following technical solution:
[0031] A speech reconstruction device, comprising:
[0032] An acoustic feature acquisition module, used to acquire acoustic features for speech signal reconstruction, wherein the acoustic features include original audio waveforms;
[0033] A first convolution processing module, used for inputting the original audio waveform into a preset first convolution processing structure, and obtaining a globally normalized audio feature representation output after being processed by the first convolution processing structure;
[0034] A second convolution processing module, used to input the globally normalized audio feature representation into a preset second convolution processing structure, and obtain audio feature representations of multiple frequency subbands output after being processed by the second convolution processing structure;
[0035] The linear processing reconstruction module is used to generate a reconstructed speech of a target length after the audio feature representations of the multiple frequency sub-bands are subjected to linear transformation processing by a preset linear processing structure.
[0036] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:
[0037] A computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the above-mentioned speech reconstruction method when executing the computer-readable instructions.
[0038] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:
[0039] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the speech reconstruction method as described above.
[0040] Compared with the prior art, the embodiments of the present application have the following beneficial effects:
[0041] The speech reconstruction method described in the embodiment of the present application obtains the acoustic features of the speech signal to be reconstructed, wherein the acoustic features include the original audio waveform; inputs the original audio waveform into a preset first convolution processing structure to obtain a globally normalized audio feature representation output after processing by the first convolution processing structure; inputs the globally normalized audio feature representation into a preset second convolution processing structure to obtain an audio feature representation of multiple frequency sub-bands output after processing by the second convolution processing structure; and generates a reconstructed speech of a target length after the audio feature representation of the multiple frequency sub-bands is subjected to linear transformation processing by a preset linear processing structure. The speech reconstruction method described in the present application improves the existing conventional convolution processing structure and linear processing results, and uses the improved first convolution processing structure, the second convolution processing structure and the linear processing structure to perform speech reconstruction, thereby reducing the amount of speech reconstruction calculations and improving the speech reconstruction efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0043] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;
[0044] Figure 2 is a flow chart of an embodiment of a speech reconstruction method according to the present application;
[0045] Figure 3 yes Figure 2 A flowchart of a specific embodiment of step 202 is shown;
[0046] Figure 4 yes Figure 3 A flowchart of a specific embodiment of step 302 is shown;
[0047] Figure 5 yes Figure 2 A flowchart of a specific embodiment of step 203 is shown;
[0048] Figure 6 yes Figure 5 A flowchart of a specific embodiment of step 502 is shown;
[0049] Figure 7 yes Figure 2 A flowchart of a specific embodiment of step 204 is shown;
[0050] Figure 8 is a structural schematic diagram of an embodiment of a speech reconstruction device according to the present application;
[0051] Fig. 9 is a structural diagram of a specific embodiment of the first convolution processing module 802 of the present application;
[0052] Fig.10 It is a structural diagram of a specific embodiment of the convolution block convolution submodule 902 described in the present application;
[0053] Fig.11 is a structural diagram of a specific embodiment of the second convolution processing module 803 of the present application;
[0054] Fig.12 is a structural diagram of a specific embodiment of the linear processing reconstruction module 804 described in the present application;
[0055] Fig.13It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION
[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.
[0057] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0058] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.
[0059] like Figure 1 As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.
[0060] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.
[0061] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the laptop computer 1011, the tablet computer 1012 or the mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer and a desktop computer, etc.
[0062] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .
[0063] It should be noted that the speech reconstruction method provided in the embodiment of the present application is generally executed by a terminal device, and accordingly, the speech reconstruction device is generally arranged in the terminal device.
[0064] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.
[0065] Continue to refer Figure 2 , shows a flow chart of an embodiment of a speech reconstruction method according to the present application. The speech reconstruction method comprises the following steps:
[0066] Step 201, obtaining acoustic features for speech signal reconstruction, wherein the acoustic features include original audio waveforms;
[0067] Specifically, the acoustic features of the speech signal to be reconstructed are obtained so that the speech reconstruction can be performed subsequently according to the acoustic features.
[0068] Step 202, inputting the original audio waveform into a preset first convolution processing structure to obtain a globally normalized audio feature representation output after being processed by the first convolution processing structure;
[0069] In this embodiment, the preset first convolution processing structure includes a first conventional convolution layer and N convolution blocks, each of which is stacked with a depth layered convolution layer, a linear normalization layer, a second conventional convolution layer, a GELU activation layer, a third conventional convolution layer and a global normalization layer in a sequential processing relationship. All convolution layers in the first convolution processing structure use a non-upsampling method when performing convolution operations, and all convolution layers have independent convolution kernels.
[0070] The convolution block is designed to use deep layered convolution to reduce the parameters of the entire vocoder. The linear normalization layer is to maintain the robustness of the vocoder. The second and third conventional convolution layers are to expand the receptive field of the vocoder so that the current output is related to a value with a longer width. Finally, the global normalization layer is used to enhance the robustness of the vocoder and prevent gradient explosion.
[0071] Specifically, the first conventional convolution layer, the depth-layered convolution layer, the second conventional convolution layer, and the third conventional convolution layer all use non-upsampling methods when performing convolution operations, and all have independent convolution kernels. The purpose of using non-upsampling methods is to reduce the amount of calculation when performing convolution operations on acoustic features; and independent convolutions realize the extraction of acoustic features of different dimensions according to different convolution kernels.
[0072] Step 203: input the globally normalized audio feature representation into a preset second convolution processing structure to obtain audio feature representations of multiple frequency subbands output after being processed by the second convolution processing structure;
[0073] In this embodiment, the preset second convolution processing structure includes a fourth conventional convolution layer and a multi-frequency subband convolution layer (Multi-Frequency Subband Convolutional Layer), and the multi-frequency subband convolution layer includes M convolution layers of different frequency subbands. The multi-frequency subband convolution layer is a specially designed convolution layer in the convolutional neural network (CNN), which is mainly used to process input data with different frequency characteristics, especially in the fields of signal processing and audio processing. The multi-frequency subband convolution layer decomposes the input data into multiple frequency subbands and performs convolution operations independently on each subband, so that features in different frequency ranges can be more effectively extracted and utilized. This design enables the network to focus more on processing information of specific frequencies, improving the performance and efficiency of the declarator.
[0074] Specifically, the present application improves the performance and efficiency of the vocoder to a certain extent by adopting a multi-frequency sub-band convolutional layer, thereby improving the efficiency of speech reconstruction.
[0075] Step 204: subject the audio feature representations of the plurality of frequency sub-bands to linear transformation processing using a preset linear processing structure to generate reconstructed speech of a target length.
[0076] In this embodiment, the preset linear processing structure includes a conventional linear layer and a reconstruction linear layer. The conventional linear layer directly performs conventional linearization processing on the audio feature representations of the multiple frequency subbands, and the reconstruction purpose of the reconstruction linear layer is to align the signal length of the audio features obtained based on the acoustic features with the target reconstructed speech to be generated, that is, to align the signal length of the audio representations of the multiple frequency subbands subjected to conventional linearization processing with the target reconstructed speech to be generated, so as to generate reconstructed speech of the target length.
[0077] Specifically, the audio feature representations of the multiple frequency subbands are first subjected to a linear transformation process using a conventional linear layer, and the results of the first linear transformation are subjected to a secondary linear transformation process using a reconstruction linear layer. Since the reconstruction linear layer has limited the length of the linear transformation result, reconstructed speech of the target length can be reconstructed.
[0078] In this embodiment, the acoustic features of the speech signal to be reconstructed are obtained, wherein the acoustic features include the original audio waveform; the original audio waveform is input into a preset first convolution processing structure to obtain a globally normalized audio feature representation output after being processed by the first convolution processing structure; the globally normalized audio feature representation is input into a preset second convolution processing structure to obtain an audio feature representation of multiple frequency sub-bands output after being processed by the second convolution processing structure; the audio feature representation of the multiple frequency sub-bands is subjected to linear transformation processing by a preset linear processing structure to generate a reconstructed speech of a target length. The speech reconstruction method described in the present application improves the existing conventional convolution processing structure and linear processing results, and adopts the improved first convolution processing structure, the second convolution processing structure and the linear processing structure to perform speech reconstruction, which not only reduces the amount of speech reconstruction calculations, but also improves the speech reconstruction efficiency.
[0079] Continue to refer Figure 3 , in some optional implementations, Figure 3 yes Figure 2 The flowchart of a specific embodiment of step 202 shown includes the following steps:
[0080] Step 301, inputting the original audio waveform into the first conventional convolutional layer, performing audio feature extraction, and obtaining a first audio feature matrix;
[0081] Step 302: Process the first audio feature matrix through N layers of convolution blocks to obtain a globally normalized audio feature representation, where N is a positive integer.
[0082] By successively subjecting the original audio waveform to the first conventional convolution layer and the N-layer convolution block processing, a globally normalized audio feature representation is obtained to facilitate subsequent conversion processing.
[0083] Continue to refer Figure 4 , in some optional implementations, Figure 4 yes Figure 3 The flowchart of a specific embodiment of step 302 shown includes the following steps:
[0084] Step 401, using the deep hierarchical convolutional layer to extract low-dimensional audio features, local audio features and global audio features from the audio features included in the first audio feature matrix in sequence;
[0085] Step 402, normalizing the extracted low-dimensional audio features, local audio features, and global audio features respectively in combination with the linear normalization layer to obtain normalized processing results;
[0086] Step 403, performing conventional convolution and nonlinear introduction operations on the normalized processing result according to the second conventional convolution layer and the GELU activation layer to obtain an activation representation result;
[0087] Step 404: Use the third conventional convolution layer to perform a conventional convolution operation on the activation representation result again, and introduce the convolution operation result into the global normalization layer for global normalization processing to obtain the globally normalized audio feature representation.
[0088] The purpose of using deep layered convolution is to reduce the parameters of the entire vocoder. The linear normalization layer is to maintain the robustness of the vocoder. The second and third regular convolution layers are to expand the receptive field of the vocoder and make the current output related to values of a longer width. Finally, the global normalization layer is used to enhance the robustness of the vocoder and prevent gradient explosion.
[0089] Continue to refer Figure 5 , in some optional implementations, Figure 5 yes Figure 2 The flowchart of a specific embodiment of step 203 shown includes the following steps:
[0090] Step 501, inputting the globally normalized audio feature representation into the fourth conventional convolutional layer, and obtaining a second audio feature matrix after convolution processing by the fourth conventional convolutional layer;
[0091] Step 502: input the second audio feature matrix into the multi-frequency sub-band convolution layer, decompose the second audio feature matrix through the multi-frequency sub-band convolution layer, and obtain audio feature representations of multiple frequency sub-bands.
[0092] Continue to refer Figure 6 , in some optional implementations, Figure 6 yes Figure 5 The flowchart of a specific embodiment of step 502 shown includes the following steps:
[0093] Step 601, using the convolution layers of the M different frequency sub-bands to respectively perform convolution operations on the second audio feature matrix to obtain audio feature convolution results respectively corresponding to the convolution layers of the M different frequency sub-bands, where M is a positive integer;
[0094] Step 602: Using the audio feature convolution results corresponding to the convolution layers of the M different frequency sub-bands as audio feature representations of multiple frequency sub-bands obtained by decomposing the second audio feature matrix.
[0095] A multi-frequency sub-band convolution layer is used to decompose the input data, that is, the second audio feature matrix, into multiple frequency sub-bands, and a convolution operation is performed independently on each sub-band, so that features in different frequency ranges can be extracted and utilized more effectively. This design enables the network to focus more on processing information of specific frequencies, improve the performance and efficiency of the declarer, and thus improve the efficiency of speech reconstruction.
[0096] Continue to refer Figure 7 , in some optional implementations, Figure 7 yes Figure 2 The flowchart of a specific embodiment of step 204 shown includes the following steps:
[0097] Step 701, using the conventional linear layer to perform conventional linear transformation processing on the audio feature representations of the multiple frequency sub-bands to obtain a linear transformation processing result;
[0098] Step 702: Perform a reconstructed linear transformation process on the linear transformation processing result by using the reconstructed linear layer to generate a reconstructed speech signal of a target length, wherein the reconstruction purpose of the reconstructed linear layer is:
[0099] The audio features obtained according to the acoustic features are aligned with the target reconstructed speech to be generated.
[0100] The audio feature representations of the multiple frequency subbands are first subjected to a linear transformation process using a conventional linear layer, and the results of the first linear transformation are subjected to a secondary linear transformation process using a reconstruction linear layer. Since the reconstruction linear layer has limited the length of the linear transformation result, reconstructed speech of the target length can be reconstructed.
[0101] In this embodiment, restrictively, all convolution layers use a non-upsampling method when performing convolution operations, and all convolution layers have independent convolution kernels, that is, the above-mentioned first conventional convolution layer, depth-layered convolution layer, second conventional convolution layer, third conventional convolution layer, fourth conventional convolution layer and multi-frequency sub-band convolution layer all use a non-upsampling method when performing convolution operations, and all have independent convolution kernels.
[0102] When performing convolution operations on acoustic features, non-upsampling methods are used, and each has an independent convolution kernel. The purpose of using non-upsampling methods is to reduce the amount of calculation during the convolution operation when performing convolution operations on acoustic features; and independent convolution realizes the extraction of acoustic features of different dimensions according to different convolution kernels. By reducing the amount of calculation during the convolution operation and extracting acoustic features of different dimensions according to different convolution kernels, the speech reconstruction efficiency of the vocoder is improved.
[0103] The present application obtains the acoustic features of the speech signal to be reconstructed, wherein the acoustic features include the original audio waveform; inputs the original audio waveform into a preset first convolution processing structure to obtain a globally normalized audio feature representation output after being processed by the first convolution processing structure; inputs the globally normalized audio feature representation into a preset second convolution processing structure to obtain an audio feature representation of multiple frequency sub-bands output after being processed by the second convolution processing structure; and generates a reconstructed speech of a target length after the audio feature representation of the multiple frequency sub-bands is subjected to linear transformation processing by a preset linear processing structure. The speech reconstruction method described in the present application improves the existing conventional convolution processing structure and linear processing results, and uses the improved first convolution processing structure, the second convolution processing structure and the linear processing structure to perform speech reconstruction, thereby reducing the amount of speech reconstruction calculations and improving the speech reconstruction efficiency.
[0104] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0105] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0106] In an embodiment of the present application, by obtaining the acoustic features of the speech signal to be reconstructed, wherein the acoustic features include the original audio waveform; the original audio waveform is input into a preset first convolution processing structure to obtain a globally normalized audio feature representation output after being processed by the first convolution processing structure; the globally normalized audio feature representation is input into a preset second convolution processing structure to obtain an audio feature representation of multiple frequency sub-bands output after being processed by the second convolution processing structure; the audio feature representation of the multiple frequency sub-bands is subjected to linear transformation processing by a preset linear processing structure to generate a reconstructed speech of a target length. The speech reconstruction method described in the present application improves the existing conventional convolution processing structure and linear processing results, and adopts the improved first convolution processing structure, the second convolution processing structure and the linear processing structure to perform speech reconstruction, which not only reduces the amount of speech reconstruction calculations, but also improves the speech reconstruction efficiency.
[0107] Further references Figure 8 , as a response to the above Figure 2 In order to realize the method shown in the figure, the present application provides an embodiment of a speech reconstruction device, and the device embodiment is Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.
[0108] like Figure 8 As shown, the speech reconstruction device 800 described in this embodiment includes: an acoustic feature acquisition module 801, a first convolution processing module 802, a second convolution processing module 803 and a linear processing reconstruction module 804. Among them:
[0109] An acoustic feature acquisition module 801 is used to acquire acoustic features for speech signal reconstruction, wherein the acoustic features include original audio waveforms;
[0110] A first convolution processing module 802, configured to input the original audio waveform into a preset first convolution processing structure, and obtain a globally normalized audio feature representation output after being processed by the first convolution processing structure;
[0111] A second convolution processing module 803, used to input the globally normalized audio feature representation into a preset second convolution processing structure, and obtain audio feature representations of multiple frequency sub-bands output after being processed by the second convolution processing structure;
[0112] The linear processing reconstruction module 804 is used to generate a reconstructed speech of a target length after subjecting the audio feature representations of the multiple frequency sub-bands to linear transformation processing using a preset linear processing structure.
[0113] Continue to refer Fig. 9 , Fig. 99 is a schematic diagram of a specific embodiment of the first convolution processing module 802 of the present application. The first convolution processing module 802 includes a first conventional convolution submodule 901 and a convolution block convolution submodule 902, wherein:
[0114] The first conventional convolution submodule 901 is used to input the original audio waveform into the first conventional convolution layer, perform audio feature extraction, and obtain a first audio feature matrix;
[0115] The convolution block convolution submodule 902 is used to obtain a globally normalized audio feature representation after processing the first audio feature matrix through N layers of convolution blocks, where N is a positive integer.
[0116] Continue to refer Fig.10 , Fig.10 It is a structural diagram of a specific embodiment of the convolution block convolution submodule 902 described in the present application. The convolution block convolution submodule 902 includes a depth layered convolution unit 10a, a linear normalization unit 10b, a conventional convolution activation unit 10c and a global normalization unit 10d, wherein:
[0117] A deep hierarchical convolution unit 10a, configured to extract low-dimensional audio features, local audio features, and global audio features from the audio features included in the first audio feature matrix in sequence using the deep hierarchical convolution layer;
[0118] A linear normalization unit 10b, used to perform normalization processing on the extracted low-dimensional audio features, local audio features and global audio features respectively in combination with the linear normalization layer to obtain a normalization processing result;
[0119] A conventional convolution activation unit 10c, configured to perform conventional convolution and nonlinear introduction operations on the normalization processing result according to the second conventional convolution layer and the GELU activation layer to obtain an activation representation result;
[0120] The global normalization unit 10d is used to use the third conventional convolution layer to perform a conventional convolution operation on the activation representation result again, and introduce the convolution operation result into the global normalization layer for global normalization processing to obtain the globally normalized audio feature representation.
[0121] Continue to refer Fig.11 , Fig.11 1 is a schematic diagram of a specific embodiment of the second convolution processing module 803 of the present application. The second convolution processing module 803 includes a fourth conventional convolution submodule 11a and a multi-frequency subband convolution submodule 11b, wherein:
[0122] A fourth conventional convolution submodule 11a, configured to input the globally normalized audio feature representation into the fourth conventional convolution layer, and obtain a second audio feature matrix after convolution processing by the fourth conventional convolution layer;
[0123] The multi-frequency sub-band convolution submodule 11b is used to input the second audio feature matrix into the multi-frequency sub-band convolution layer, decompose the second audio feature matrix through the multi-frequency sub-band convolution layer, and obtain audio feature representations of multiple frequency sub-bands.
[0124] Continue to refer Fig.12 , Fig.12 1 is a schematic diagram of a specific embodiment of the linear processing and reconstruction module 804 of the present application. The linear processing and reconstruction module 804 includes a conventional linear conversion submodule 12a and a reconstruction linear conversion submodule 12b, wherein:
[0125] A conventional linear conversion submodule 12a is used to perform conventional linear transformation processing on the audio feature representations of the multiple frequency subbands using the conventional linear layer to obtain a linear transformation processing result;
[0126] The reconstructed linear conversion submodule 12b is used to use the reconstructed linear layer to reconstruct the linear transformation processing result to generate a reconstructed speech signal of a target length, wherein the reconstruction purpose of the reconstructed linear layer is to align the signal length of the audio features obtained based on the acoustic features with the target reconstructed speech to be generated.
[0127] The present application obtains the acoustic features of the speech signal to be reconstructed, wherein the acoustic features include the original audio waveform; inputs the original audio waveform into a preset first convolution processing structure to obtain a globally normalized audio feature representation output after being processed by the first convolution processing structure; inputs the globally normalized audio feature representation into a preset second convolution processing structure to obtain an audio feature representation of multiple frequency sub-bands output after being processed by the second convolution processing structure; and generates a reconstructed speech of a target length after the audio feature representation of the multiple frequency sub-bands is subjected to linear transformation processing by a preset linear processing structure. The speech reconstruction method described in the present application improves the existing conventional convolution processing structure and linear processing results, and uses the improved first convolution processing structure, the second convolution processing structure and the linear processing structure to perform speech reconstruction, thereby reducing the amount of speech reconstruction calculations and improving the speech reconstruction efficiency.
[0128] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).
[0129] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.
[0130] To solve the above technical problems, the present application also provides a computer device. Fig.13 , Fig.13 This is a basic structural block diagram of the computer device in this embodiment.
[0131] The computer device 13 includes a memory 13a, a processor 13b, and a network interface 13c which are interconnected through a system bus. It should be noted that: Fig.13 Only a computer device 13 having a component memory 13a, a processor 13b, and a network interface 13c is shown, but it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.
[0132] The computer device may be a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with the user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.
[0133] The memory 13a includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 13a can be an internal storage unit of the computer device 13, such as a hard disk or memory of the computer device 13. In other embodiments, the memory 13a can also be an external storage device of the computer device 13, such as a plug-in hard disk equipped on the computer device 13, a smart memory card (Smart Med ia Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 13a can also include both the internal storage unit of the computer device 13 and its external storage device. In this embodiment, the memory 13a is generally used to store an operating system and various application software installed on the computer device 13, such as a computer-readable instruction of a speech reconstruction method, etc. In addition, the memory 13a can also be used to temporarily store various data that have been output or will be output.
[0134] The processor 13b may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor or other data processing chips in some embodiments. The processor 13b is generally used to control the overall operation of the computer device 13. In the present embodiment, the processor 13b is used to run the computer-readable instructions or process data stored in the memory 13a, such as the computer-readable instructions for running the speech reconstruction method.
[0135] The network interface 13c may include a wireless network interface or a wired network interface. The network interface 13c is generally used to establish a communication connection between the computer device 13 and other electronic devices.
[0136] The computer device proposed in this embodiment belongs to the field of speech processing technology and is applied to the scene of speech reconstruction based on acoustic features. The present application obtains the acoustic features of the speech signal to be reconstructed, wherein the acoustic features include the original audio waveform; the original audio waveform is input into the preset first convolution processing structure to obtain the globally normalized audio feature representation output after being processed by the first convolution processing structure; the globally normalized audio feature representation is input into the preset second convolution processing structure to obtain the audio feature representation of multiple frequency sub-bands output after being processed by the second convolution processing structure; the audio feature representation of the multiple frequency sub-bands is subjected to linear transformation processing by the preset linear processing structure to generate a reconstructed speech of the target length. The speech reconstruction method described in the present application improves the existing conventional convolution processing structure and linear processing results, and adopts the improved first convolution processing structure, the second convolution processing structure and the linear processing structure to perform speech reconstruction, which not only reduces the amount of speech reconstruction calculations, but also improves the speech reconstruction efficiency.
[0137] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by a processor to enable the processor to perform the steps of the speech reconstruction method as described above.
[0138] The computer-readable storage medium proposed in this embodiment belongs to the field of speech processing technology and is applied to the scene of speech reconstruction based on acoustic features. The present application obtains the acoustic features of the speech signal to be reconstructed, wherein the acoustic features include the original audio waveform; the original audio waveform is input into the preset first convolution processing structure to obtain the globally normalized audio feature representation output after being processed by the first convolution processing structure; the globally normalized audio feature representation is input into the preset second convolution processing structure to obtain the audio feature representation of multiple frequency sub-bands output after being processed by the second convolution processing structure; the audio feature representation of the multiple frequency sub-bands is subjected to linear transformation processing by the preset linear processing structure to generate a reconstructed speech of the target length. The speech reconstruction method described in the present application improves the existing conventional convolution processing structure and linear processing results, and adopts the improved first convolution processing structure, the second convolution processing structure and the linear processing structure to perform speech reconstruction, which not only reduces the amount of speech reconstruction calculations, but also improves the speech reconstruction efficiency.
[0139] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.
[0140] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.
Claims
1. A speech reconstruction method, characterized in that: The steps include: Acquiring acoustic features for speech signal reconstruction, wherein the acoustic features include original audio waveforms; Inputting the original audio waveform into a preset first convolution processing structure to obtain a globally normalized audio feature representation output after being processed by the first convolution processing structure; Inputting the globally normalized audio feature representation into a preset second convolution processing structure to obtain audio feature representations of multiple frequency subbands output after being processed by the second convolution processing structure; The audio feature representations of the multiple frequency sub-bands are subjected to linear transformation processing by a preset linear processing structure to generate reconstructed speech of a target length.
2. The speech reconstruction method according to claim 1, characterized in that: The preset first convolution processing structure includes a first conventional convolution layer and N convolution blocks. The step of inputting the original audio waveform into the preset first convolution processing structure to obtain a globally normalized audio feature representation output after being processed by the first convolution processing structure specifically includes: Inputting the original audio waveform into the first conventional convolutional layer to extract audio features and obtain a first audio feature matrix; After the first audio feature matrix is processed by N layers of convolution blocks, a globally normalized audio feature representation is obtained, where N is a positive integer.
3. The speech reconstruction method according to claim 2, characterized in that: Each of the convolution blocks is stacked with a deep hierarchical convolution layer, a linear normalization layer, a second conventional convolution layer, a GELU activation layer, a third conventional convolution layer and a global normalization layer in a sequential processing relationship. The step of obtaining a globally normalized audio feature representation after processing the first audio feature matrix by N layers of convolution blocks specifically includes: Using the deep hierarchical convolutional layer to extract low-dimensional audio features, local audio features and global audio features from the audio features included in the first audio feature matrix in sequence; In combination with the linear normalization layer, the extracted low-dimensional audio features, local audio features and global audio features are respectively normalized to obtain normalized processing results; According to the second conventional convolution layer and the GELU activation layer, performing conventional convolution and nonlinear introduction operations on the normalization processing result to obtain an activation representation result; The third conventional convolution layer is used to perform a conventional convolution operation on the activation representation result again, and the convolution operation result is introduced into the global normalization layer for global normalization processing to obtain the globally normalized audio feature representation.
4. The speech reconstruction method according to claim 1, characterized in that: The preset second convolution processing structure includes a fourth conventional convolution layer and a multi-frequency sub-band convolution layer, and the step of inputting the globally normalized audio feature representation into the preset second convolution processing structure to obtain the audio feature representations of multiple frequency sub-bands output after being processed by the second convolution processing structure specifically includes: Inputting the globally normalized audio feature representation into the fourth conventional convolutional layer, and obtaining a second audio feature matrix after convolution processing by the fourth conventional convolutional layer; The second audio feature matrix is input into the multi-frequency sub-band convolution layer, and the second audio feature matrix is decomposed by the multi-frequency sub-band convolution layer to obtain audio feature representations of multiple frequency sub-bands.
5. The speech reconstruction method according to claim 4, characterized in that: The multi-frequency sub-band convolution layer includes M convolution layers of different frequency sub-bands, and the step of inputting the second audio feature matrix into the multi-frequency sub-band convolution layer, decomposing the second audio feature matrix through the multi-frequency sub-band convolution layer, and obtaining audio feature representations of multiple frequency sub-bands specifically includes: Using the M convolution layers of different frequency sub-bands to perform convolution operations on the second audio feature matrix respectively, to obtain audio feature convolution results corresponding to the M convolution layers of different frequency sub-bands respectively, where M is a positive integer; The audio feature convolution results respectively corresponding to the convolution layers of the M different frequency sub-bands are used as audio feature representations of multiple frequency sub-bands obtained by decomposing the second audio feature matrix.
6. The speech reconstruction method according to claim 1, characterized in that: The preset linear processing structure includes a conventional linear layer and a reconstruction linear layer. The step of generating a reconstructed speech of a target length after the audio feature representations of the multiple frequency sub-bands are subjected to linear transformation processing by the preset linear processing structure specifically includes: Using the conventional linear layer, conventional linear transformation processing is performed on the audio feature representations of the multiple frequency sub-bands to obtain a linear transformation processing result; The linear transformation processing result is subjected to a reconstructed linear transformation processing by using the reconstructed linear layer to generate a reconstructed speech signal of a target length, wherein the reconstruction purpose of the reconstructed linear layer is: The audio features obtained according to the acoustic features are aligned with the target reconstructed speech to be generated.
7. The speech reconstruction method according to any one of claims 2 to 5, characterized in that: All convolutional layers use non-upsampling when performing convolution operations, and all convolutional layers have independent convolution kernels.
8. A speech reconstruction device, characterized in that: include: An acoustic feature acquisition module, used to acquire acoustic features for speech signal reconstruction, wherein the acoustic features include original audio waveforms; A first convolution processing module, used for inputting the original audio waveform into a preset first convolution processing structure, and obtaining a globally normalized audio feature representation output after being processed by the first convolution processing structure; A second convolution processing module, used to input the globally normalized audio feature representation into a preset second convolution processing structure, and obtain audio feature representations of multiple frequency subbands output after being processed by the second convolution processing structure; The linear processing reconstruction module is used to generate a reconstructed speech of a target length after the audio feature representations of the multiple frequency sub-bands are subjected to linear transformation processing by a preset linear processing structure.
9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the speech reconstruction method according to any one of claims 1 to 7 when executing the computer-readable instructions.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the speech reconstruction method according to any one of claims 1 to 7 are implemented.