Voice data processing method, device, computer equipment and storage medium
By training the speech processing model to process lossy voice data, the problem of quality degradation in voice data transmission is solved, high-quality voice data restoration is achieved, and transmission costs are reduced and processing efficiency is improved.
Patent Information
- Application Number
- CN202110660920.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-06-15
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2041-06-15
AI Technical Summary
The prior art has a degradation in the quality of lossy voice data caused by compression in voice data transmission, making it difficult to effectively restore to lossless voice data, and the existing methods are not effective.
By obtaining lossy voice data and corresponding sample voice data, the speech processing model is trained to learn the features required for speech restoration, and the speech feature processing is performed using encoding and decoding networks to achieve high-quality restoration of speech data.
It improves the processing effect of voice data, can restore higher quality voice data while transmitting low code rates, reduces network transmission costs, and improves the processing efficiency of voice data.
Smart Images

Figure CN113823296B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a method, apparatus, computer device and storage medium for processing voice data. Background Art
[0002] With the development of computer technology and network technology, voice data can be transmitted between different computer devices. In order to improve transmission efficiency, the voice data is usually compressed by the sender at a lower bit rate and then sent to the receiver. However, compressing voice data will cause the quality of voice data to decrease, resulting in lossy voice data received by the receiver.
[0003] In order to solve the above problems, the receiving end can filter the received lossy speech data to extract the spectral features of the speech data in the frequency domain, and then restore the lossless speech data based on the spectral features, but the effect of speech restoration is still not good enough. Summary of the invention
[0004] The embodiments of the present application provide a method, apparatus, computer device and storage medium for processing voice data, which can improve the effect of voice data processing. The technical solution is as follows:
[0005] In one aspect, a method for processing speech data is provided, the method comprising:
[0006] Acquire lossy voice data and sample voice data corresponding to the lossy voice data, wherein the lossy voice data is obtained by compressing the sample voice data;
[0007] Calling a speech processing model, encoding the lossy speech data to obtain speech features, decoding the speech features to obtain target speech data having a quality parameter higher than that of the lossy speech data, wherein the quality parameter represents a degree of quality loss of the speech data;
[0008] The speech processing model is trained according to the difference between the sample speech data and the target speech data.
[0009] In another aspect, a speech data processing device is provided, the device comprising:
[0010] A data acquisition module, used to acquire lossy voice data and sample voice data corresponding to the lossy voice data, wherein the lossy voice data is obtained by compressing the sample voice data;
[0011] A speech processing module, configured to call a speech processing model, encode the lossy speech data to obtain speech features, decode the speech features, and obtain target speech data having a quality parameter higher than that of the lossy speech data, wherein the quality parameter represents a degree of quality loss of the speech data;
[0012] The model training module is used to train the speech processing model according to the difference between the sample speech data and the target speech data.
[0013] Optionally, the speech processing model includes an encoding network and a decoding network, the encoding network includes m encoding layers, the decoding network includes n decoding layers, the m and the n are integers greater than 1, and the speech processing module includes:
[0014] An encoding unit, configured to call a k-th encoding layer to encode the lossy speech data to obtain a k-th encoded speech feature, where k is a positive integer less than m;
[0015] The encoding unit is further used to continue calling the next encoding layer of the k-th encoding layer, encode the k-th encoded speech feature, and obtain the next encoded speech feature, until the speech feature output by the last encoding layer is obtained;
[0016] A decoding unit, used for calling the h-th decoding layer to decode the speech feature to obtain the h-th decoded speech feature, where h is a positive integer less than n;
[0017] The decoding unit is also used to continue calling the next decoding layer of the h-th decoding layer, decode the h-th decoded speech feature, and obtain the next decoded speech feature until the target speech data output by the last decoding layer is obtained.
[0018] Optionally, the lossy speech data includes a plurality of sampling points, and the speech processing module includes:
[0019] The sampling point encoding unit is used to call the speech processing model, encode the multiple sampling points in parallel to obtain the speech features, and decode the speech features to obtain the target speech data.
[0020] Optionally, the voice processing module includes:
[0021] A sub-band division unit, configured to divide the lossy speech data into sub-bands to obtain a plurality of lossy sub-band data, each lossy sub-band data belonging to a different frequency band;
[0022] A sub-band data encoding unit, configured to call the speech processing model, encode the plurality of lossy sub-band data to obtain the speech features, and decode the speech features to obtain a plurality of target sub-band data;
[0023] The sub-band merging unit is used to perform sub-band merging on the multiple target sub-band data to obtain the target speech data.
[0024] Optionally, the sub-band merging unit is used to perform sub-band merging on the multiple target sub-band data, and perform smoothing on the speech data obtained by the sub-band merging to obtain the target speech data.
[0025] Optionally, the model training module includes:
[0026] A mel spectrum extraction unit, used to perform mel spectrum extraction on the sample speech data and the target speech data respectively, to obtain sample mel spectrum features corresponding to the sample speech data and target mel spectrum features corresponding to the target speech data;
[0027] The first training unit is used to train the speech processing model according to the difference between the sample speech data and the target speech data, and the difference between the sample Mel-spectrogram feature and the target Mel-spectrogram feature.
[0028] Optionally, the device further comprises:
[0029] A speech discrimination module, used for calling the speech discrimination model, discriminating the target speech data, and obtaining a first discrimination result, wherein the first discrimination result is used for indicating whether the target speech data is generated by calling the speech processing model;
[0030] The model training module includes:
[0031] The second training unit is used to train the speech processing model according to the difference between the sample speech data and the target speech data and the first discrimination result.
[0032] Optionally, the speech discrimination module is further used to call the speech discrimination model to discriminate the sample speech data to obtain a second discrimination result, wherein the second discrimination result is used to indicate whether the sample speech data is generated by calling the speech processing model;
[0033] The model training module is also used to train the speech discrimination model according to the first discrimination result and the second discrimination result.
[0034] Optionally, the model training module includes:
[0035] The third training unit is used to train the speech processing model according to the difference between the sample speech data and the target speech data, the first discrimination result, and the difference between the first discrimination result and the second discrimination result.
[0036] Optionally, the speech discrimination model includes a frequency domain discrimination model and a time domain discrimination model, the first discrimination result includes a third discrimination result output by the frequency domain discrimination model and a fourth discrimination result output by the time domain discrimination model, and the speech discrimination module includes:
[0037] A frequency domain transform unit, used for performing frequency domain transform on the target speech data to obtain frequency domain speech data;
[0038] A first discrimination unit, configured to call the frequency domain discrimination model to discriminate the frequency domain speech data to obtain the third discrimination result;
[0039] The second discrimination unit is used to call the time domain discrimination model to discriminate the target speech data to obtain the fourth discrimination result.
[0040] Optionally, the speech processing module is further used to call the trained speech processing model, encode any speech data to obtain a first speech feature, and decode the first speech feature to obtain speech data having a quality parameter higher than that of any speech data.
[0041] On the other hand, a computer device is provided, which includes a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the operations performed in the voice data processing method described in the above aspects.
[0042] On the other hand, a computer-readable storage medium is provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement the operations performed in the voice data processing method as described in the above aspects.
[0043] On the other hand, a computer program product or a computer program is provided, wherein the computer program product or the computer program includes a computer program code, wherein the computer program code is stored in a computer-readable storage medium, and a processor of a computer device reads the computer program code from the computer-readable storage medium, and the processor executes the computer program code, so that the computer device implements the operations performed in the speech data processing method described in the above aspects.
[0044] The method, apparatus, computer equipment and storage medium provided in the embodiments of the present application use lossy speech data and corresponding sample speech data to train a speech processing model. The sample speech data is lossless speech data before compression, and the lossy speech data is lossy speech data after compression. Therefore, the speech processing model can learn the features required for speech restoration, thereby providing a method for repairing speech data based on the speech processing model, which can improve the processing effect of speech data. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.
[0046] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present application.
[0047] Figure 2 It is a structural diagram of a speech processing model provided in an embodiment of the present application.
[0048] Figure 3 It is a structural diagram of a speech processing model provided in an embodiment of the present application.
[0049] Figure 4 It is a structural schematic diagram of a residual convolution provided in an embodiment of the present application.
[0050] Figure 5 It is a flow chart of a voice data processing method provided in an embodiment of the present application.
[0051] Figure 6 It is a flow chart of a voice data processing method provided in an embodiment of the present application.
[0052] Figure 7 It is a structural diagram of a frequency domain discrimination model provided in an embodiment of the present application.
[0053] Figure 8 It is a structural diagram of a time domain discrimination model provided in an embodiment of the present application.
[0054] Fig. 9 It is a schematic diagram of a voice data processing result provided in an embodiment of the present application.
[0055] Fig.10 It is a schematic diagram of another voice data processing result provided in an embodiment of the present application.
[0056] Fig.11It is a structural diagram of a voice data processing device provided in an embodiment of the present application.
[0057] Fig.12 It is a structural diagram of another voice data processing device provided in an embodiment of the present application.
[0058] Fig.13 It is a structural diagram of a terminal provided in an embodiment of the present application.
[0059] Fig.14 It is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION
[0060] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.
[0061] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of this application, a first coding layer may be referred to as a second coding layer, and similarly, a second coding layer may be referred to as a first coding layer.
[0062] Here, at least one refers to one or more than one, for example, at least one sampling point can be one sampling point, two sampling points, three sampling points, or any other integer greater than or equal to one. Multiple refers to two or more than two, for example, multiple sampling points can be two sampling points, three sampling points, or any other integer greater than or equal to two. Each refers to each of at least one, for example, each sampling point refers to each sampling point of the multiple sampling points. If the multiple sampling points are 3 sampling points, each sampling point refers to each sampling point of the 3 sampling points.
[0063] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.
[0064] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics, etc. Artificial intelligence software technologies include natural language processing technology and machine learning.
[0065] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.
[0066] The key technologies of speech technology include automatic speech recognition technology (ASR, Automatic Speech Recognition), speech synthesis technology (TTS, Text To Speech) and voiceprint recognition technology. Enabling computers to listen, see, speak and feel is the future development direction of human-computer interaction, among which speech has become one of the most promising human-computer interaction methods in the future.
[0067] The following will explain the voice data processing method provided in the embodiment of the present application based on artificial intelligence technology and voice technology.
[0068] The voice data processing method provided in the embodiment of the present application can be used in a computer device. Optionally, the computer device is a terminal or a server. Optionally, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Optionally, the terminal is a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited to this.
[0069] In one possible implementation, the computer program involved in the embodiments of the present application may be deployed and executed on a computer device, or on multiple computer devices located at one location, or on multiple computer devices distributed at multiple locations and interconnected through a communication network. Multiple computer devices distributed at multiple locations and interconnected through a communication network can constitute a blockchain system.
[0070] In one possible implementation, the computer device used to train the speech processing model in the embodiment of the present application is a node in the blockchain system, which can store the trained speech processing model in the blockchain, and then the node or the node corresponding to other devices in the blockchain can process the speech data based on the speech processing model.
[0071] Figure 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application, see Figure 1 , the implementation environment includes: a terminal 101 and a server 102. The terminal 101 and the server 102 are connected via a wireless or wired network. Optionally, the server 102 is used to train a speech processing model using the method provided in the embodiment of the present application, and the speech processing model is used to process the speech data and restore higher quality speech data. The server 102 sends the trained speech processing model to the terminal 101, and the terminal 101 can call the speech processing model to process the speech data to obtain processed speech data.
[0072] In a possible implementation, an application client provided by the server is run in the terminal 101, and the server 102 stores the trained speech processing model in the application client, and the application client has functions such as speech transmission and speech processing. Based on the application client, the terminal 101 calls the speech processing model, processes the speech data, and obtains the processed speech data.
[0073] It should be noted that Figure 1 In the description, only the example of the server 102 training the speech processing model and sending it to the terminal 101 is used for explanation. In another embodiment, the speech processing model can also be directly trained by the terminal 101.
[0074] The voice data processing method provided in the embodiments of the present application can be applied to any scenario of processing voice data.
[0075] Scenarios for transmitting voice data, such as voice call scenarios, video call scenarios, voice conference scenarios, and video conference scenarios. The first terminal collects original voice data. In order to improve the transmission efficiency of voice data, the first terminal encodes the original voice data and sends the encoded voice features to the second terminal. The second terminal decodes the received voice features to obtain lossy voice data. Compared with the original voice data, the lossy voice data loses some information. The second terminal calls the voice processing model to process the lossy voice data to restore the target voice data. Compared with the lossy voice data, the target voice data restored by this processing has higher quality.
[0076] Figure 2 This is a schematic diagram of the structure of a speech processing model provided in an embodiment of the present application, see Figure 2 , the speech processing model includes a coding network 201 and a decoding network 202. The coding network 201 is used to encode speech data to obtain speech features, and the decoding network 202 is used to decode speech features to obtain speech data. Optionally, the coding network 201 includes a plurality of coding layers 211, wherein every two adjacent coding layers 211 in the plurality of coding layers 211 are interconnected, and each coding layer 211 has the same network structure but different network parameters, and the plurality of coding layers 211 can extract deep-level features of speech data. Optionally, the decoding network 202 includes a plurality of decoding layers 212, wherein every two adjacent decoding layers 212 in the plurality of decoding layers 212 are interconnected, and each decoding layer 212 has the same network structure but different network parameters, and the plurality of decoding layers 212 can decode deep-level features.
[0077] In a possible implementation, the structure diagram of the speech processing model is as follows: Figure 3As shown. The encoding network 201 of the speech processing model includes N encoding layers 211, where N is an integer greater than 1. Every two adjacent encoding layers 211 in the N encoding layers 211 are connected to each other, and the output of the encoding layer 211 is used as the input of the next encoding layer 211, and the output of the last encoding layer 211 is used as the output of the encoding network 201. Each encoding layer 211 includes a downsampling convolution and multiple parallel residual convolutions, and the output of each encoding layer 211 is the sum of the outputs of multiple residual convolutions. The decoding network 202 of the speech processing model includes N decoding layers 212, where every two adjacent decoding layers 212 in the N decoding layers 212 are connected to each other, and the output of the decoding layer 212 is used as the input of the next decoding layer 212, and the output of the last decoding layer 212 is used as the output of the decoding network 202. Each decoding layer 212 includes an upsampling convolution and multiple parallel residual convolutions, and the output of each decoding layer 212 is the sum of the outputs of multiple residual convolutions. Optionally, the speech processing model also includes a plurality of one-dimensional convolution network structures. Optionally, the structural diagram of the residual convolution is as follows Figure 4 As shown, each residual convolution is composed of 3 groups of dilated convolutions and one-dimensional convolutions.
[0078] The function of the speech processing model is to process the speech data decoded by the receiving device to restore higher quality speech data. The embodiment of the present application provides a speech data processing method based on the speech processing model by training the speech processing model. The training process of the speech processing model is described in detail below. Figure 5 and Figure 6 Embodiment of the invention.
[0079] Figure 5 is a flow chart of a method for processing voice data provided by an embodiment of the present application. The execution subject of the embodiment of the present application is a computer device. Optionally, the computer device is the above-mentioned Figure 1 The terminal or server in the embodiment of . Figure 5 , the method comprising:
[0080] 501. A computer device obtains lossy voice data and sample voice data corresponding to the lossy voice data.
[0081] In order to train the speech processing model, the computer device first obtains a sample training set, which includes a plurality of lossy speech data and sample speech data corresponding to the lossy speech data. The process of training the speech processing model based on the sample training set includes multiple iterations, and in each iteration, training is performed based on a pair of lossy speech data and sample speech data corresponding to the lossy speech data. It should be noted that steps 501-503 in the embodiment of the present application are only described by taking one iteration process as an example.
[0082] Among them, the lossy speech data is obtained by compressing the sample speech data, and the compression includes encoding and / or quantization and other processing. Since the compression process will lose part of the information in the speech data, the sample speech data can be regarded as the original lossless speech data, and the lossy speech data can be regarded as the compressed lossy speech data.
[0083] 502. The computer device calls a speech processing model, encodes the lossy speech data to obtain speech features, and decodes the speech features to obtain target speech data having a higher quality parameter than the lossy speech data.
[0084] The computer device stores a speech processing model. The structure of the speech processing model is described in detail above. Figure 2-4 Embodiment. The computer device obtains the lossy speech data and the corresponding sample speech data in the sample data set, and calls the speech processing model to encode the lossy speech data to obtain speech features, and the speech features are used to describe the features of the lossy speech data. The computer device calls the speech processing model, and continues to decode the speech features to obtain target speech data with a quality parameter higher than the lossy speech data, and the quality parameter represents the quality loss degree of the speech data, and the quality parameter is negatively correlated with the quality loss degree of the speech data.
[0085] 503. The computer device trains a speech processing model based on the difference between the sample speech data and the target speech data.
[0086] Since the goal of the speech processing model is to process the damaged speech data so that the processed target speech data is closer to the sample speech data, thereby improving the quality of the speech data. Therefore, the more similar the target speech data is to the sample speech data, the more accurate the speech processing model is. The computer device trains the speech processing model based on the difference between the sample speech data and the target speech data to improve the processing capability of the speech processing model, thereby improving the accuracy of the speech processing model.
[0087] The method provided in the embodiment of the present application uses lossy speech data and corresponding sample speech data to train a speech processing model. The sample speech data is lossless speech data before compression, and the lossy speech data is lossy speech data after compression. Therefore, the speech processing model can learn the features required for speech restoration, thereby providing a method for repairing speech data based on the speech processing model, which can improve the processing effect of speech data.
[0088] Figure 6 This is a flow chart of a voice data processing method provided by an embodiment of the present application. The execution subject of the embodiment of the present application is a computer device, see Figure 6 , the method comprising:
[0089] 601. A computer device obtains lossy voice data and sample voice data corresponding to the lossy voice data.
[0090] Here, step 601 is the same as the above step 501 and will not be described in detail here.
[0091] 602. The computer device calls the coding network in the speech processing model to encode the lossy speech data to obtain speech features.
[0092] The computer device stores a speech processing model. The structure of the speech processing model is described in detail above. Figure 2-4 The speech processing model includes a coding network, which is used to encode speech data to obtain speech features. After the computer device obtains the lossy speech data, it calls the coding network in the speech processing model to encode the sample speech data to obtain speech features, and the speech features are used to represent the features of the lossy speech data.
[0093] In a possible implementation, the speech feature includes long-term speech features and short-term speech features of the speech data, the long-term speech feature includes the human voice features of the speech data, the human voice features belong to the information of the speaker, and the short-term speech feature includes the rhythm features and content features of the speech data, etc. The speech feature is a high-dimensional hidden layer feature. In addition, the speech feature may also include other types of speech features, which are not limited in the embodiments of the present application.
[0094] In a possible implementation, the coding network includes m coding layers, where m is an integer greater than 1, and every two adjacent coding layers in the m coding layers are connected to each other. The computer device calls the kth coding layer to encode the lossy speech data to obtain the kth coded speech feature, where k is a positive integer less than m; and continues to call the next coding layer of the kth coding layer to encode the kth coded speech feature to obtain the next coded speech feature, until the speech feature output by the last coding layer is obtained.
[0095] Among them, the input of the entire coding network is used as the input of the first coding layer. For each coding layer after the first coding layer, the output of the previous coding layer is used as the input of the current coding layer. The output of the last coding layer is the output of the entire coding network, which is the speech feature obtained by encoding the lossy speech data.
[0096] Optionally, the network structure of each coding layer is the same, but the network parameters are different, for example, the convolution kernel size of the residual convolution in each coding layer is different, and the multiple coding layers are multiple coding layers of different scales. The different scales of the multiple coding layers refer to different downsampling multiples of the multiple coding layers. For example, the downsampling multiples of the multiple coding layers are arranged in order from small to large according to the multiple coding layers. This enables the encoding of speech data from precise to coarse through multiple coding layers of different scales, and can learn high-dimensional speech features in the speech data, so as to facilitate the subsequent restoration of high-quality speech data based on the high-dimensional speech features.
[0097] 603. The computer device calls a decoding network in the speech processing model to decode the speech features to obtain target speech data having a higher quality parameter than the lossy speech data.
[0098] The speech processing model also includes a decoding network, which is used to decode the speech features. After the computer device obtains the speech features, it calls the decoding network in the speech processing model to decode the speech features to obtain target speech data with a quality parameter higher than that of the lossy speech data.
[0099] Among them, the quality parameter characterizes the degree of quality loss of the voice data, and the quality parameter is negatively correlated with the degree of quality loss of the voice data. The larger the quality parameter of the voice data, the smaller the degree of quality loss of the voice data, and the smaller the quality parameter of the voice data, the greater the degree of loss of the voice data. Since the lossy voice data is compressed voice data, the compression process causes the lossy voice data to lose some features. In the embodiment of the present application, the lost features are modeled through a voice processing model, and the quality parameters of the voice data are determined based on these features. In the training stage of the model, the voice processing model learns how to supplement the lost features through continuous training, thereby improving the quality parameters of the voice data, so that the quality parameters of the target voice data are closer and closer to the quality parameters of the uncompressed voice data, and avoid the target voice data decoded by the computer device deviating from the features satisfied by the uncompressed voice data.
[0100] For example, the features lost by the compressed voice data belong to the auditory perception features, and the quality parameter is determined based on the auditory perception features, which are used to describe the perception of the human ear to the voice data, so the quality parameter can measure the auditory perception of the human ear to the voice data. The embodiment of the present application decodes the voice features to obtain the target voice data with the auditory perception features, so that the auditory perception of the target voice data by the human ear is closer and closer to the auditory perception of the uncompressed voice data by the human ear, thereby improving the quality parameters of the target voice data. For example, in order to test the perception of the human ear to the voice data, the voice data is scored by MUSHRA (Multi-Stimulus Test with Hidden Reference and Anchor, multi-stimulus hidden reference benchmark test method), the score of the uncompressed voice data is 95 points, and the score of the lossy voice data is 60 points. Then, after calling the voice processing model to process the lossy voice data, the target voice data with a quality parameter higher than the lossy voice data is obtained, so that the score of the target voice data is close to 95 points.
[0101] In the embodiment of the present application, the sample speech data is the original speech data, and the lossy speech data is the speech data obtained after compressing the sample speech data. The purpose of training the speech processing model is to make the target speech data output by the speech processing model closer and closer to the sample speech data, thereby achieving the effect of restoring the sample speech data from the lossy speech data, thereby improving the accuracy of the speech processing model.
[0102] In a possible implementation, the decoding network includes n decoding layers, where n is an integer greater than 1, and every two adjacent decoding layers in the n decoding layers are connected to each other. The computer device calls the h-th decoding layer to decode the speech feature to obtain the h-th decoded speech feature, where h is a positive integer less than n; and continues to call the next decoding layer of the h-th decoding layer to decode the h-th decoded speech feature to obtain the next decoded speech feature, until the target speech data output by the last decoding layer is obtained.
[0103] Among them, the input of the entire decoding network is used as the input of the first decoding layer. For each decoding layer after the first decoding layer, the output of the previous decoding layer is used as the input of the current decoding layer. The output of the last decoding layer is the output of the entire decoding network, which is also the target speech data.
[0104] Optionally, the network structure of each decoding layer is the same, but the network parameters are different, for example, the convolution kernel size of the dilated convolution in each decoding layer is different, and the multiple decoding layers are multiple decoding layers of different scales. The different scales of the multiple decoding layers refer to different upsampling multiples of the multiple decoding layers. For example, the upsampling multiples of the multiple decoding layers are arranged from small to large in the order of the multiple decoding layers, thereby achieving decoding of the voice data from precise to coarse through multiple decoding layers of different scales, thereby restoring high-quality voice data.
[0105] In the embodiment of the present application, the lossy speech data is the input of the speech processing model, and the target speech data is the output of the speech processing model. The speech processing model can be regarded as an end-to-end speech processing model. Since the lossy speech data is the only input of the end-to-end speech processing model during the processing, the speech processing model does not rely on other auxiliary features or fundamental frequency information of the speech data during the processing, and the speech processing model has good robustness.
[0106] It should be noted that, by executing the above steps 602-603, the computer device implements calling the speech processing model, encoding the lossy speech data to obtain speech features, decoding the speech features, and obtaining target speech data having a quality parameter higher than the lossy speech data.
[0107] In a possible implementation, the lossy speech data includes a plurality of sampling points. The plurality of sampling points are sampling points obtained by sampling a continuous speech signal, and the sampling points are used to represent the signal strength of the speech signal at the sampling position. The computer device calls the speech processing model, encodes the plurality of sampling points in parallel, obtains speech features, decodes the speech features, and obtains target speech data with a quality parameter higher than the lossy speech data. Encoding the plurality of sampling points in parallel means encoding the plurality of sampling points at the same time, and outputting a plurality of target feature vectors at the same time. The plurality of target feature vectors are speech features, and then the plurality of target feature vectors are decoded in parallel, that is, the plurality of target feature vectors are decoded at the same time, and the target speech data is output, and the target speech data also includes a plurality of sampling points.
[0108] Optionally, the speech processing model in the embodiment of the present application is a fully convolutional neural network structure that can be processed in parallel. Compared with the autoregressive method in the related art, the parallel method provided by the embodiment of the present application has higher processing efficiency for speech processing. In addition, it can also avoid the situation where the error accumulation caused by the use of the autoregressive method, thereby improving the accuracy of speech processing.
[0109] In one possible implementation, a computer device divides the lossy speech data into sub-bands to obtain multiple lossy sub-band data, calls a speech processing model, encodes the multiple lossy sub-band data to obtain speech features, decodes the speech features to obtain multiple target sub-band data, and performs sub-band merging on the multiple target sub-band data to obtain target speech data.
[0110] Each lossy sub-band data belongs to a different frequency band, and the computer device divides the frequency of the lossy voice data into multiple frequency bands, and the lossy voice data of each frequency band is used as a lossy sub-band data. Since the embodiment of the present application divides the voice data into multiple different sub-band data for separate processing, the length of the time series of the voice data can be reduced, which is conducive to speeding up the processing process of the voice processing model and improving the processing efficiency of the voice processing model.
[0111] Among them, the lossy sub-band data belongs to the time domain data, the lossy voice data includes multiple sampling points, and each lossy sub-band data also includes multiple sampling points. For example, the lossy voice data includes 16000*1 sampling points, that is, 16000 1-dimensional sampling points, and the lossy voice data is divided into 4 lossy sub-band data, each lossy sub-band data includes 4000 1-dimensional sampling points. The computer device merges the 4 lossy sub-band data into a matrix as a whole, calls the voice processing model, and encodes the 4 lossy sub-band data. It can be understood that after the 4 lossy sub-band data are merged into a matrix, they include 4000*4 sampling points, that is, 4000 4-dimensional sampling points. The computer device can use parallel processing to process each 4-dimensional sampling point, thereby realizing the processing of the 4 lossy sub-band data.
[0112] Optionally, the computer device uses PQMF (Pseudo quadratue mirror filter) to perform sub-band division on the lossy speech data to obtain multiple lossy sub-band data. Subsequently, the PQMF is also used to perform sub-band merging on multiple target sub-band data to obtain target speech data.
[0113] Optionally, the computer device performs sub-band merging on multiple target sub-band data, and performs smoothing on the voice data obtained by the sub-band merging to obtain the target voice data. Figure 3 As shown, in order to avoid the frequency discontinuity caused by boundary problems when the target sub-band data is merged, a one-dimensional convolution network structure can be deployed after PQMF to smooth the merged speech data to obtain the target speech data after smoothing.
[0114] 604. The computer device calls the speech discrimination model to discriminate the target speech data to obtain a first discrimination result.
[0115] Through the above-mentioned speech processing model, the target speech data that can be restored from the lossy speech data, but the target speech data may contain potential noise. In order to further improve the sound quality of the speech data, the embodiment of the present application uses the idea of adversarial training to model the distribution characteristics of the speech data. Among them, the speech discrimination model is used to distinguish whether the speech data is generated by calling the speech processing model, the speech data that has not been processed by the speech processing model is real speech data, and the speech data generated by the speech processing model can be regarded as "forged" speech data. The purpose of the speech discrimination model is to distinguish whether the speech data is real speech data or "forged" speech data. Optionally, the speech discrimination model is GAN (Generative adversarial network, adversarial generation network). The computer device obtains the target speech data, then calls the speech discrimination model, discriminates the target speech data, and obtains a first discrimination result, which is used to indicate whether the target speech data is generated by calling the speech processing model.
[0116] Since the target voice data is voice data generated according to the voice features, it is actually forged voice data, not real sample voice data. Moreover, the more accurate the voice features are, the more similar the forged target voice data is to the real sample voice data. Therefore, the voice discrimination model can be called to discriminate the target voice data to determine whether the target voice data is the voice data generated by the voice processing model. If the voice discrimination model successfully discriminates that the target voice data is generated by the voice processing model, it means that the processing capacity of the voice processing model is insufficient, resulting in inaccurate voice features extracted, or inaccurate target voice data generated by the voice processing model. If the voice discrimination model does not discriminate that the target voice data is generated by the voice processing model, it means that the voice processing model has sufficient processing capacity, so the extracted voice features and the generated target voice data are relatively accurate. Therefore, the model parameters of the voice processing model can be adjusted according to the discrimination result of the voice discrimination model to make the voice processing model more accurate.
[0117] In one possible implementation, the speech discrimination model includes a frequency domain discrimination model and a time domain discrimination model, and the first discrimination result includes a third discrimination result output by the frequency domain discrimination model and a fourth discrimination result output by the time domain discrimination model. The computer device performs frequency domain transformation on the target speech data to obtain frequency domain speech data, calls the frequency domain discrimination model, discriminates the frequency domain speech data, obtains the third discrimination result, calls the time domain discrimination model, discriminates the target speech data, and obtains the fourth discrimination result. Among them, the frequency domain discrimination model can be one or more, and the time domain discrimination model can be one or more.
[0118] Optionally, the number of frequency domain discrimination models is multiple, and the number of time domain discrimination models is multiple. The computer device calls multiple frequency domain discrimination models, and discriminates the frequency domain speech data respectively to obtain multiple third discrimination results, and subsequently trains the speech processing model according to the multiple third discrimination results. For example, the multiple frequency domain discrimination models are frequency domain discrimination models of different resolutions, and the computer device performs multiple frequency domain transformations on the target speech data to obtain multiple frequency domain speech data of different resolutions, and then calls the frequency domain discrimination model to discriminate the frequency domain speech data of the same resolution to obtain multiple third discrimination results. The computer device calls multiple time domain discrimination models, and discriminates the target speech data respectively to obtain multiple fourth discrimination results, and subsequently trains the speech processing model according to the multiple fourth discrimination results. For example, the multiple time domain discrimination models are time domain discrimination models of different scales, and the computer device performs pooling processing on the target speech data to obtain multiple time domain speech data of different scales, and then calls the time domain discrimination model to discriminate the time domain speech data of the same scale to obtain multiple fourth discrimination results.
[0119] Figure 7 This is a schematic diagram of the structure of a frequency domain discrimination model provided in an embodiment of the present application, see Figure 7 Each frequency domain discrimination model includes 1 two-dimensional convolution and 3 convolution blocks, and each convolution block includes downsampling two-dimensional convolution and two-dimensional convolution. The computer device performs multiple short-time Fourier transforms on the target speech data to obtain multiple frequency domain speech data with different resolutions, and inputs them into the corresponding frequency domain discrimination model for discrimination.
[0120] Figure 8 This is a schematic diagram of the structure of a time domain discrimination model provided in an embodiment of the present application, see Figure 8 Each time domain discrimination model includes 3 one-dimensional convolutions and 1 group one-dimensional convolution. The computer device performs multiple pooling processes on the target speech data to obtain multiple time domain speech data of different scales, and inputs them into the corresponding time domain discrimination model for discrimination.
[0121] 605. The computer device trains a speech processing model according to the difference between the sample speech data and the target speech data and the first discrimination result.
[0122] Since the sample voice data is the original voice data, and the target voice data is the voice data obtained by restoring the compressed lossy voice data, the more similar the target voice data is to the sample voice data, the more accurate the voice processing model is. The first discrimination result is used to indicate whether the target voice data is generated by calling the voice processing model. If the first discrimination result indicates that the target voice data is not generated by calling the voice processing model, the more accurate the voice processing model is; if the first discrimination result indicates that the target voice data is generated by calling the voice processing model, the less accurate the voice processing model is.
[0123] In a possible implementation, the first discrimination result is a first discrimination probability, which indicates the probability that the target speech data is not generated by calling the speech processing model. The computer device updates the network parameters of the speech processing model according to the difference between the sample speech data and the target speech data and the first discrimination probability, so that the difference between the sample speech data and the target speech data becomes smaller and smaller, and the first discrimination probability becomes larger and larger, thereby improving the accuracy of the speech processing model.
[0124] In one possible implementation, the computer device calls the speech discrimination model to discriminate the sample speech data to obtain a second discrimination result, where the second discrimination result is used to indicate whether the sample speech data is generated by calling the speech processing model. The computer device trains the speech discrimination model based on the first discrimination result and the second discrimination result.
[0125] Among them, the target voice data is forged voice data, and the sample voice data is real voice data. If the first discrimination result indicates that the target voice data is generated by calling the voice processing model, and the second discrimination result indicates that the sample voice data is not generated by calling the voice processing model, it means that the voice discrimination result can accurately distinguish whether the voice data is generated by calling the voice processing model. If the first discrimination result indicates that the target voice data is not generated by calling the voice processing model, or the second discrimination result indicates that the sample voice data is generated by calling the voice processing model, it means that the voice discrimination result cannot accurately distinguish whether the voice data is generated by calling the voice processing model. Therefore, the computer device can train the voice discrimination model according to the first discrimination result and the second discrimination result, so that the voice discrimination model can accurately distinguish whether the voice data is generated by calling the voice processing model. Optionally, the first discrimination result is a first discrimination probability, which indicates the probability that the target speech data is not generated by calling the speech processing model; the second discrimination result is a second discrimination probability, which indicates the probability that the sample speech data is not generated by calling the speech processing model. The computer device updates the network parameters of the speech discrimination model according to the first discrimination probability and the second discrimination probability, so that the first discrimination probability becomes smaller and smaller and the second discrimination probability becomes larger and larger, thereby improving the accuracy of the speech discrimination model.
[0126] During the training process, the higher the accuracy of the speech discrimination model, the higher the requirement for the accuracy of the speech processing model, thereby prompting the continuous improvement of the accuracy of the speech processing model. The higher the accuracy of the speech processing model, the higher the requirement for the accuracy of the speech discrimination model, thereby prompting the continuous improvement of the accuracy of the speech discrimination model, thereby forming a virtuous cycle of mutual confrontation, so that the accuracy of the trained speech processing model is getting higher and higher. The embodiment of the present application proposes to use adversarial training to improve the repair effect of damaged speech data. Based on the above-mentioned speech processing model, an additional speech discrimination model is constructed to model the distribution characteristics of speech data, thereby improving the quality of speech data processing.
[0127] Optionally, the computer device uses the following formula to obtain a first loss value, and updates the network parameters of the speech discrimination model according to the first loss value.
[0128]
[0129] in, represents the first loss value, represents lossy speech data, x represents sample speech data, G(·) represents the speech processing model, represents the target speech data, D s represents the sth speech discrimination model, represents the mathematical expectation, and ||·||1 represents the first norm.
[0130] In another possible implementation, the computer device trains the speech processing model according to the difference between the sample speech data and the target speech data, the first discrimination result, and the difference between the first discrimination result and the second discrimination result.
[0131] The target voice data is voice data obtained by restoring the compressed lossy voice data, which is actually forged voice data, not real voice data, and the sample voice data is real voice data. The smaller the difference between the first discrimination result and the second discrimination result obtained by the voice discrimination model for discriminating the sample voice data and the target voice data respectively, the smaller the difference between the target voice data and the sample voice data, that is, the more accurate the target voice data is. Therefore, the computer device can also train the voice processing model according to the difference between the first discrimination result and the second discrimination result, so that the difference between the first discrimination result and the second discrimination result becomes smaller and smaller, thereby improving the accuracy of the voice processing model.
[0132] It should be noted that the embodiment of the present application uses the voice discrimination model for adversarial training as an example. In another embodiment, the voice discrimination model may not be used for adversarial training, and the above step 604 does not need to be performed. The computer device directly trains the voice processing model based on the difference between the sample voice data and the target voice data.
[0133] In another possible implementation, the computer device performs Mel-spectrogram extraction on the sample speech data and the target speech data respectively, obtains the sample Mel-spectrogram features corresponding to the sample speech data, and the target Mel-spectrogram features corresponding to the target speech data, and trains the speech processing model according to the difference between the sample speech data and the target speech data, and the difference between the sample Mel-spectrogram features and the target Mel-spectrogram features. The Mel-spectrogram feature is a two-dimensional feature optimized for human auditory perception, which retains perceptual information. The speech processing model is trained by the difference between the sample Mel-spectrogram features and the target Mel-spectrogram features, so that the training process takes into account the difference between the auditory perception of the sample speech data and the auditory perception of the target speech data, which is conducive to enabling the speech processing model to learn the features of auditory perception.
[0134] Optionally, the computer device uses the following formula to obtain a target loss value and updates the network parameters of the speech processing model according to the target loss value.
[0135]
[0136] Among them, L represents the target loss value, λ1, λ2 and λ3 represent the weight parameters corresponding to each loss value, represents the second loss value, Represents the third loss value, L F Represents the fourth loss value.
[0137] The formula for the second loss value is as follows:
[0138]
[0139] in, represents lossy speech data, x represents sample speech data, G(·) represents the speech processing model, represents the target speech data, φ(·) represents the Mel spectrum extraction process, represents the mathematical expectation, and ||·||1 represents the first norm.
[0140] The formula for the third loss value is as follows:
[0141]
[0142] in, represents the target speech data, G(·) represents the speech processing model, represents the target speech data, D s represents the sth speech discrimination model, represents mathematical expectation.
[0143] The formula for the fourth loss value is as follows:
[0144]
[0145] in, represents lossy speech data, x represents sample speech data, G(·) represents the speech processing model, represents the target speech data, represents the output of the i-th layer network in the s-th speech discrimination model, N represents the number of speech discrimination models, T represents the number of layers of the network in the speech discrimination model, represents the mathematical expectation, and ||·||1 represents the first norm.
[0146] It should be noted that the above steps 601-605 are only described by taking one iteration process as an example. In the process of training the speech processing model, multiple iterations are required. In one possible implementation, in response to the iteration reaching the first threshold, the speech processing model is stopped from being trained; or, in response to the loss value obtained in the current iteration round being not greater than the second threshold, the speech processing model is stopped from being trained. The first threshold and the second threshold are both arbitrary values, for example, the first threshold is 10 or 15, and the second threshold is 0.4 or 0.3, etc.
[0147] The goal of the speech encoder and speech decoder is to compress the speech data at a certain bit rate on the sending device, and then transmit it to the receiving device through the network for decoding, thereby restoring the speech data. However, when the speech data is encoded at a lower bit rate, it is difficult for the receiving device to restore high-quality speech data.
[0148] The embodiment of the present application proposes an end-to-end lossy voice data restoration method, which can restore the voice data generated after passing through a low-bit-rate voice encoder and a voice decoder into lossless voice data. Based on the voice processing model provided by the embodiment of the present application, it is possible to restore high-quality voice data when transmitting low-bit-rate voice data without changing the codec framework, which is compatible with different codec frameworks and can reduce the cost of voice data transmission.
[0149] The method provided in the embodiment of the present application uses lossy speech data and corresponding sample speech data to train a speech processing model. The sample speech data is lossless speech data before compression, and the lossy speech data is lossy speech data after compression. Therefore, the speech processing model can learn the features required for speech restoration, thereby providing a method for repairing speech data based on the speech processing model, which can improve the processing effect of speech data.
[0150] Furthermore, encoding multiple sampling points in parallel enables processing of multiple information at the same time, which is beneficial to improving the processing efficiency of voice data processing.
[0151] Furthermore, dividing the speech data into a plurality of different sub-band data for separate processing can reduce the length of the time series of the speech data, which is beneficial to speeding up the processing process of the speech processing model and improving the processing efficiency of the speech processing model.
[0152] Moreover, during the training process, the higher the accuracy of the speech discrimination model, the higher the requirement for the accuracy of the speech processing model, which in turn prompts the continuous improvement of the accuracy of the speech processing model. The higher the accuracy of the speech processing model, the higher the requirement for the accuracy of the speech discrimination model, which in turn prompts the continuous improvement of the accuracy of the speech discrimination model, thus forming a virtuous circle of mutual confrontation and making the accuracy of the trained speech processing model higher and higher.
[0153] Above Figure 5 and Figure 6 The speech processing model trained in the embodiment can be applied to the task of repairing speech data. The computer device calls the trained speech processing model, encodes any speech data to obtain a first speech feature, decodes the first speech feature, and obtains speech data with a quality parameter higher than that of any speech data. The any speech data is the speech data to be repaired, for example, the lossy speech data obtained after the receiving end device decodes the received speech feature. The speech data with a quality parameter higher than that of any speech data is the repaired speech data, and the quality of the repaired speech data is higher than that of the speech data before the repair, thereby achieving the repair of any speech data to obtain high-quality speech data.
[0154] In order to verify the speech data processing method provided in the embodiment of the present application, the repair effect of the damaged speech data obtained by speech encoding and decoding at different bit rates was evaluated. First, the objective index PESQ (Perceptual evaluation of speech quality) of the model was measured. The higher the PESQ, the better the processing effect of the model. The results are shown in Table 1 below, where Opus represents the unrepaired damaged speech data, and Opus-GAN (Opus-Generative Adversarial Networks) represents the speech data obtained by repairing the damaged speech data.
[0155] Table 1
[0156]
[0157] From Table 1, it can be seen that under different bit rate conditions, the speech processing model proposed in the embodiment of the present application can significantly improve the effect of the speech data obtained by Opus direct encoding and decoding under low bit rate conditions, which means that when the network transmission conditions are poor, even if Opus works in a low bit rate encoding mode, the receiving end device can obtain high-quality speech data through the speech processing model. In addition, under the encoding and decoding conditions of 6kb / s, the effect of the speech processing model proposed in the embodiment of the application even exceeds the encoding and decoding effect under 10kb / s, so the quality of the speech data can be guaranteed while reducing the transmission bit rate of the network, thereby saving network transmission costs.
[0158] In order to further measure the effect of the speech processing model proposed in the embodiment of the present application, a supervisor audiometry was organized, and the quality of the speech data restored by different speech encoders was scored using the MUSHRA (Multi-Stimulus Test with Hidden Reference and Anchor) method, with a full score of 100. The results are as follows: Fig. 9 As shown. Fig. 9 It can be found that the score of the original voice data without compression is more than 95 points, while the score of the voice data without voice restoration at a low bit rate is less than 40 points under the condition of 6kb / s, which means that the compressed voice data will greatly lose the quality of the voice data and even damage the intelligibility of the voice data. However, by using the voice processing model of the embodiment of the present application, the quality of the restored voice data has been significantly improved, and a result of 80 points was obtained at a bit rate of 6kb / s, which means that the voice processing model of the embodiment of the present application can greatly improve the quality of voice data, thereby restoring high-quality voice data in lossy voice data.
[0159] Since the above experiments are based on speech synthesis data recorded by a high-fidelity microphone, in order to further verify it on real speech data, it is necessary to build corresponding speech processing models for different bit rates and a speech processing model for mixed bit rates. First, the objective indicator PESQ is obtained. The results are shown in Table 2 below:
[0160] Table 2
[0161]
[0162] In Table 2, Opus-GAN is a speech processing model related to the bit rate, and Opus-GAN-mix is a speech processing model independent of the bit rate. From the results in Table 2, it can be seen that the speech processing model of the embodiment of the present application can still well repair the sound quality of the damaged speech data. In addition, the Opus-GAN-mix model can also improve the sound quality of the damaged speech data, and the objective indicator PESQ of the Opus-GAN-mix model is slightly worse than that of the Opus-GAN model.
[0163] Similarly, MUSHRA scoring was performed for the processing effects of different bit rates. The results are as follows: Fig.10 As shown. Fig.10 It can be found that due to the low quality of the test data, the score of the original voice data without compression is only about 90 points, and the score of the voice data without voice restoration at a low bit rate is less than 30 points at 6kb / s. When the voice processing model of the embodiment of the present application is added, the quality of the restored voice data is significantly improved, and a score close to 70 is obtained at a bit rate of 6kb / s. For the Opus-GAN-mix model, its MUSHRA score is only slightly lower than that of the Opus-GAN model. The above results show that the voice processing model of the embodiment of the present application can repair compressed voice data of different qualities, thereby achieving the purpose of improving transmission sound quality and reducing transmission bit rate.
[0164] Fig.11 is a structural diagram of a voice data processing device provided in an embodiment of the present application. Fig.11 , the device comprises:
[0165] The data acquisition module 1101 is used to acquire lossy speech data and sample speech data corresponding to the lossy speech data, where the lossy speech data is obtained by compressing the sample speech data;
[0166] The speech processing module 1102 is used to call the speech processing model, encode the lossy speech data to obtain speech features, decode the speech features, and obtain target speech data having a quality parameter higher than that of the lossy speech data, wherein the quality parameter represents a degree of quality loss of the speech data;
[0167] The model training module 1103 is used to train the speech processing model according to the difference between the sample speech data and the target speech data.
[0168] The speech data processing device provided in the embodiment of the present application uses lossy speech data and corresponding sample speech data to train a speech processing model. The sample speech data is lossless speech data before compression, and the lossy speech data is lossy speech data after compression. Therefore, the speech processing model can learn the features required for speech restoration, thereby providing a method for repairing speech data based on the speech processing model, which can improve the processing effect of speech data.
[0169] Alternatively, see Fig.12 The speech processing model includes a coding network and a decoding network, the coding network includes m coding layers, the decoding network includes n decoding layers, m and n are integers greater than 1, and the speech processing module 1102 includes:
[0170] The encoding unit 1112 is used to call the k-th encoding layer to encode the lossy speech data to obtain the k-th encoded speech feature, where k is a positive integer less than m;
[0171] The encoding unit 1112 is further used to continue calling the next encoding layer of the k-th encoding layer, encode the k-th encoded speech feature, and obtain the next encoded speech feature, until the speech feature output by the last encoding layer is obtained;
[0172] The decoding unit 1122 is used to call the h-th decoding layer to decode the speech feature to obtain the h-th decoded speech feature, where h is a positive integer less than n;
[0173] The decoding unit 1122 is further used to continue calling the next decoding layer of the h-th decoding layer, decode the h-th decoded speech feature, and obtain the next decoded speech feature, until the target speech data output by the last decoding layer is obtained.
[0174] Alternatively, see Fig.12 The lossy speech data includes a plurality of sampling points. The speech processing module 1102 includes:
[0175] The sampling point encoding unit 1132 is used to call the speech processing model, encode multiple sampling points in parallel to obtain speech features, and decode the speech features to obtain target speech data.
[0176] Alternatively, see Fig.12 , the speech processing module 1102 includes:
[0177] A sub-band division unit 1142 is used to divide the lossy speech data into sub-bands to obtain a plurality of lossy sub-band data, each lossy sub-band data belonging to a different frequency band;
[0178] The sub-band data encoding unit 1152 is used to call the speech processing model, encode the multiple lossy sub-band data to obtain speech features, and decode the speech features to obtain multiple target sub-band data;
[0179] The sub-band merging unit 1162 is used to perform sub-band merging on multiple target sub-band data to obtain target speech data.
[0180] Optionally, the sub-band merging unit 1162 is configured to perform sub-band merging on a plurality of target sub-band data, and perform smoothing processing on the speech data obtained by the sub-band merging to obtain target speech data.
[0181] Alternatively, see Fig.12 , the model training module 1103 includes:
[0182] The mel spectrum extraction unit 1113 is used to perform mel spectrum extraction on the sample speech data and the target speech data respectively to obtain sample mel spectrum features corresponding to the sample speech data and target mel spectrum features corresponding to the target speech data;
[0183] The first training unit 1123 is used to train the speech processing model according to the difference between the sample speech data and the target speech data, and the difference between the sample mel-spectrogram feature and the target mel-spectrogram feature.
[0184] Alternatively, see Fig.12 , the device further comprises:
[0185] The speech discrimination module 1104 is used to call the speech discrimination model to discriminate the target speech data and obtain a first discrimination result, wherein the first discrimination result is used to indicate whether the target speech data is generated by calling the speech processing model;
[0186] The model training module 1103 includes:
[0187] The second training unit 1133 is used to train the speech processing model according to the difference between the sample speech data and the target speech data and the first discrimination result.
[0188] Alternatively, see Fig.12 The speech discrimination module 1104 is further used to call the speech discrimination model to discriminate the sample speech data to obtain a second discrimination result, where the second discrimination result is used to indicate whether the sample speech data is generated by calling the speech processing model;
[0189] The model training module 1103 is also used to train the speech discrimination model according to the first discrimination result and the second discrimination result.
[0190] Alternatively, see Fig.12 , the model training module 1103 includes:
[0191] The third training unit 1143 is used to train the speech processing model according to the difference between the sample speech data and the target speech data, the first discrimination result, and the difference between the first discrimination result and the second discrimination result.
[0192] Alternatively, see Fig.12 The speech discrimination model includes a frequency domain discrimination model and a time domain discrimination model, the first discrimination result includes a third discrimination result output by the frequency domain discrimination model and a fourth discrimination result output by the time domain discrimination model, and the speech discrimination module 1104 includes:
[0193] The frequency domain transform unit 1114 is used to perform frequency domain transform on the target speech data to obtain frequency domain speech data;
[0194] The first discrimination unit 1124 is used to call the frequency domain discrimination model to discriminate the frequency domain speech data to obtain a third discrimination result;
[0195] The second discrimination unit 1134 is used to call the time domain discrimination model to discriminate the target speech data to obtain a fourth discrimination result.
[0196] Alternatively, see Fig.12 The speech processing module 1102 is also used to call the trained speech processing model, encode any speech data to obtain a first speech feature, decode the first speech feature, and obtain speech data with a quality parameter higher than that of any speech data.
[0197] It should be noted that: the voice data processing device provided in the above embodiment only uses the division of the above functional modules as an example to illustrate when processing voice data. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the voice data processing device provided in the above embodiment and the voice data processing method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.
[0198] An embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the operations performed in the voice data processing method of the above embodiment.
[0199] Optionally, the computer device is provided as a terminal. Fig.13 A schematic diagram of the structure of a terminal 1300 provided by an exemplary embodiment of the present application is shown.
[0200] The terminal 1300 includes a processor 1301 and a memory 1302 .
[0201] The processor 1301 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1301 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). The processor 1301 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1301 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1301 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.
[0202] The memory 1302 may include one or more computer-readable storage media, which may be non-transitory. The memory 1302 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1302 is used to store at least one computer program, which is used to be possessed by the processor 1301 to implement the voice data processing method provided in the method embodiment of the present application.
[0203] In some embodiments, the terminal 1300 may further optionally include: a peripheral device interface 1303 and at least one peripheral device. The processor 1301, the memory 1302 and the peripheral device interface 1303 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1303 via a bus, a signal line or a circuit board. Optionally, the peripheral device includes: at least one of a radio frequency circuit 1304, a display screen 1305, a camera assembly 1306, an audio circuit 1307 and a power supply 1309.
[0204] The peripheral device interface 1303 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1301 and the memory 1302. In some embodiments, the processor 1301, the memory 1302, and the peripheral device interface 1303 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1301, the memory 1302, and the peripheral device interface 1303 may be implemented on a separate chip or circuit board, which is not limited in this embodiment.
[0205] The radio frequency circuit 1304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1304 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1304 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 1304 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 1304 can communicate with other devices through at least one wireless communication protocol. The wireless communication protocol includes, but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1304 may also include circuits related to NFC (Near Field Communication), which is not limited in this application.
[0206] The display screen 1305 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1305 is a touch display screen, the display screen 1305 also has the ability to collect touch signals on the surface or above the surface of the display screen 1305. The touch signal can be input to the processor 1301 as a control signal for processing. At this time, the display screen 1305 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 1305 can be one, set on the front panel of the terminal 1300; in other embodiments, the display screen 1305 can be at least two, respectively set on different surfaces of the terminal 1300 or in a folding design; in other embodiments, the display screen 1305 can be a flexible display screen, set on a curved surface or a folding surface of the terminal 1300. Even, the display screen 1305 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 1305 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).
[0207] The camera assembly 1306 is used to capture images or videos. Optionally, the camera assembly 1306 includes a front camera and a rear camera. The front camera is arranged on the front panel of the terminal 1300, and the rear camera is arranged on the back of the terminal 1300. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1306 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.
[0208] The audio circuit 1307 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 1301 for processing, or input them into the radio frequency circuit 1304 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 1300. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 1301 or the radio frequency circuit 1304 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into sound waves audible to humans, but also convert the electrical signal into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1307 may also include a headphone jack.
[0209] The power supply 1309 is used to power various components in the terminal 1300. The power supply 1309 can be an alternating current, a direct current, a disposable battery, or a rechargeable battery. When the power supply 1309 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.
[0210] In some embodiments, the terminal 1300 further includes one or more sensors 1310 , including but not limited to: an acceleration sensor 1311 , a gyroscope sensor 1312 , a pressure sensor 1313 , an optical sensor 1315 , and a proximity sensor 1316 .
[0211] The acceleration sensor 1311 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal 1300. For example, the acceleration sensor 1311 can be used to detect the components of gravity acceleration on the three coordinate axes. The processor 1301 can control the display screen 1305 to display the user interface in a horizontal view or a vertical view according to the gravity acceleration signal collected by the acceleration sensor 1311. The acceleration sensor 1311 can also be used to collect motion data of games or users.
[0212] The gyro sensor 1312 can detect the body direction and rotation angle of the terminal 1300, and the gyro sensor 1312 can cooperate with the acceleration sensor 1311 to collect the user's 3D actions on the terminal 1300. The processor 1301 can implement the following functions based on the data collected by the gyro sensor 1312: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.
[0213] The pressure sensor 1313 can be set on the side frame of the terminal 1300 and / or the lower layer of the display screen 1305. When the pressure sensor 1313 is set on the side frame of the terminal 1300, it can detect the user's holding signal of the terminal 1300, and the processor 1301 performs left and right hand recognition or shortcut operation according to the holding signal collected by the pressure sensor 1313. When the pressure sensor 1313 is set on the lower layer of the display screen 1305, the processor 1301 controls the operability controls on the UI interface according to the user's pressure operation on the display screen 1305. The operability controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.
[0214] The optical sensor 1315 is used to collect the ambient light intensity. In one embodiment, the processor 1301 can control the display brightness of the display screen 1305 according to the ambient light intensity collected by the optical sensor 1315. Optionally, when the ambient light intensity is high, the display brightness of the display screen 1305 is increased; when the ambient light intensity is low, the display brightness of the display screen 1305 is decreased. In another embodiment, the processor 1301 can also dynamically adjust the shooting parameters of the camera assembly 1306 according to the ambient light intensity collected by the optical sensor 1315.
[0215] The proximity sensor 1316, also called a distance sensor, is disposed on the front panel of the terminal 1300. The proximity sensor 1316 is used to collect the distance between the user and the front of the terminal 1300. In one embodiment, when the proximity sensor 1316 detects that the distance between the user and the front of the terminal 1300 is gradually decreasing, the processor 1301 controls the display screen 1305 to switch from the screen-on state to the screen-off state; when the proximity sensor 1316 detects that the distance between the user and the front of the terminal 1300 is gradually increasing, the processor 1301 controls the display screen 1305 to switch from the screen-off state to the screen-on state.
[0216] Those skilled in the art will understand that Fig.13 The structure shown in the figure does not constitute a limitation on the terminal 1300, and the terminal 1300 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.
[0217] Optionally, the computer device is provided as a server. Fig.14It is a structural diagram of a server provided in an embodiment of the present application. The server 1400 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1401 and one or more memories 1402, wherein the memory 1402 stores at least one computer program, and the at least one computer program is loaded and executed by the processor 1401 to implement the methods provided in the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the server may also include other components for implementing device functions, which will not be described in detail here.
[0218] An embodiment of the present application also provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement the operations performed in the voice data processing method of the above embodiment.
[0219] The embodiment of the present application also provides a computer program product or computer program, the computer program product or computer program includes a computer program code, the computer program code is stored in a computer-readable storage medium, the processor of the computer device reads the computer program code from the computer-readable storage medium, and the processor executes the computer program code, so that the computer device implements the operations performed in the voice data processing method of the above embodiment. In some embodiments, the computer program involved in the embodiment of the present application can be deployed on a computer device for execution, or on multiple computer devices located at one location, or on multiple computer devices distributed at multiple locations and interconnected by a communication network. Multiple computer devices distributed at multiple locations and interconnected by a communication network can constitute a blockchain system.
[0220] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.
[0221] The above description is only an optional embodiment of the embodiments of the present application and is not intended to limit the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the present application.
Claims
1. A method for processing speech data, characterized in that: The method comprises: Acquire lossy speech data and sample speech data corresponding to the lossy speech data, wherein the lossy speech data is obtained by compressing the sample speech data, and the lossy speech data is obtained by compressing through any codec framework, and the features lost in the compression process of the lossy speech data belong to auditory perception features, and the auditory perception features are used to describe the auditory perception of the speech data by the human ear; Dividing the lossy speech data into sub-bands to obtain a plurality of lossy sub-band data, each lossy sub-band data belonging to a different frequency band, the lossy sub-band data including a plurality of sampling points, the plurality of sampling points being sampling points obtained by sampling a continuous speech signal, the sampling points being used to represent the signal strength of the speech signal at a sampling position; Inputting the plurality of lossy sub-band data into an end-to-end speech processing model; the speech processing model comprises a coding network and a decoding network, the coding network comprises m coding layers, the decoding network comprises n decoding layers, wherein m and n are integers greater than 1; the m coding layers have different downsampling multiples, and the m coding layers are arranged in ascending order according to the downsampling multiples; the n decoding layers have different upsampling multiples, and the n decoding layers are arranged in ascending order according to the upsampling multiples; The speech processing model encodes multiple sampling points in the lossy sub-band data in parallel for each lossy sub-band data in the multiple lossy sub-band data to obtain multiple target feature vectors; wherein the encoding process includes: calling the kth encoding layer in the speech processing model to encode the lossy sub-band data to obtain the kth encoding target feature vector, where k is a positive integer less than m; the target feature vector includes long-term speech features and short-term speech features of the speech data, the long-term speech features include the human voice features of the speech data, and the short-term speech features include the rhythm features and content features of the speech data; continuing to call the next encoding layer of the kth encoding layer to encode the kth encoding target feature vector to obtain the next encoding target feature vector, until the target feature vector output by the last encoding layer is obtained; Decoding the multiple target feature vectors in parallel to obtain target sub-band data; wherein the decoding process includes: calling the h-th decoding layer to decode the target feature vector to obtain the h-th decoded target feature vector, where h is a positive integer less than n; continuing to call the next decoding layer of the h-th decoding layer to decode the h-th decoded target feature vector to obtain the next decoded target feature vector, until the target sub-band data output by the last decoding layer is obtained; Merging the obtained multiple target sub-band data into sub-bands, smoothing the speech data obtained by merging the sub-bands, and outputting target speech data having a quality parameter higher than that of the lossy speech data, wherein the quality parameter represents the degree of quality loss of the speech data, and the multiple lossy sub-band data are the only inputs of the speech processing model during the processing; Performing multiple frequency domain transformations on the target speech data to obtain multiple frequency domain speech data with different resolutions, calling multiple frequency domain discrimination models with different resolutions, and discriminating the frequency domain speech data with the same resolution respectively to obtain multiple third discrimination results, wherein the third discrimination results are used to indicate whether the target speech data is generated by calling the speech processing model; Performing multiple pooling processes on the target speech data to obtain multiple time-domain speech data of different scales, calling multiple time-domain discrimination models of different scales, and discriminating the time-domain speech data of the same scale respectively to obtain multiple fourth discrimination results, wherein the fourth discrimination results are used to indicate whether the target speech data is generated by calling the speech processing model; respectively calling the frequency domain discrimination model and the time domain discrimination model to discriminate the sample speech data to obtain a second discrimination result, wherein the second discrimination result is used to indicate whether the sample speech data is generated by calling the speech processing model; Performing mel-spectrogram extraction on the sample speech data and the target speech data respectively to obtain sample mel-spectrogram features corresponding to the sample speech data and target mel-spectrogram features corresponding to the target speech data; wherein the mel-spectrogram features are features optimized for auditory perception of human ears; The speech processing model is trained according to the difference between the sample speech data and the target speech data, the multiple third discrimination results and the multiple fourth discrimination results, the difference between the sample Mel-spectrogram feature and the target Mel-spectrogram feature, and the difference between the multiple third discrimination results and the multiple fourth discrimination results and the second discrimination result respectively.
2. The method according to claim 1, characterized in that The method further comprises: A speech discrimination model is trained according to the plurality of third discrimination results, the plurality of fourth discrimination results and the second discrimination result.
3. The method according to claim 1, characterized in that After training the speech processing model according to the difference between the sample speech data and the target speech data, the plurality of third discrimination results and the plurality of fourth discrimination results, the difference between the sample Mel spectrum feature and the target Mel spectrum feature, and the difference between the plurality of third discrimination results and the plurality of fourth discrimination results and the second discrimination result, the method further comprises: The trained speech processing model is called to encode any speech data to obtain a first speech feature, and the first speech feature is decoded to obtain speech data having a quality parameter higher than that of any speech data.
4. A voice data processing device, characterized in that: The device comprises: A data acquisition module, used for acquiring lossy speech data and sample speech data corresponding to the lossy speech data, wherein the lossy speech data is obtained by compressing the sample speech data, and the lossy speech data is obtained by compressing through any coding and decoding framework, and the features lost in the compression process of the lossy speech data belong to auditory perception features, and the auditory perception features are used to describe the auditory perception of the human ear to the speech data; A speech processing module, configured to divide the lossy speech data into sub-bands to obtain a plurality of lossy sub-band data, each lossy sub-band data belonging to a different frequency band, the lossy sub-band data comprising a plurality of sampling points, the plurality of sampling points being sampling points obtained by sampling a continuous speech signal, the sampling points being used to represent the signal strength of the speech signal at the sampling position; The speech processing module is also used to input the multiple lossy sub-band data into an end-to-end speech processing model; the speech processing model includes a coding network and a decoding network, the coding network includes m coding layers, the decoding network includes n decoding layers, and m and n are integers greater than 1; the m coding layers have different downsampling multiples, and the m coding layers are arranged in ascending order according to the downsampling multiples; the n decoding layers have different upsampling multiples, and the n decoding layers are arranged in ascending order according to the upsampling multiples; the speech processing model encodes multiple sampling points in the lossy sub-band data in parallel for each lossy sub-band data in the multiple lossy sub-band data to obtain multiple target feature vectors; wherein the encoding process includes: calling the kth coding layer in the speech processing model to encode the lossy sub-band data to obtain the kth encoding target feature vector, and k is a positive integer less than m. number; the target feature vector includes long-term speech features and short-term speech features of speech data, the long-term speech features include vocal features of speech data, and the short-term speech features include rhythmic features and content features of speech data; continue to call the next encoding layer of the k-th encoding layer, encode the k-th encoding target feature vector, and obtain the next encoding target feature vector, until the target feature vector output by the last encoding layer is obtained; decode the multiple target feature vectors in parallel to obtain target sub-band data; wherein the decoding process includes: calling the h-th decoding layer, decoding the target feature vector, and obtaining the h-th decoding target feature vector, wherein h is a positive integer less than the n; continue to call the next decoding layer of the h-th decoding layer, decode the h-th decoding target feature vector, and obtain the next decoding target feature vector, until the target sub-band data output by the last decoding layer is obtained; Merging the obtained multiple target sub-band data into sub-bands, smoothing the speech data obtained by merging the sub-bands, and outputting target speech data having a quality parameter higher than that of the lossy speech data, wherein the quality parameter represents the degree of quality loss of the speech data, and the multiple lossy sub-band data are the only inputs of the speech processing model during the processing; A speech discrimination module, used for performing multiple frequency domain transformations on the target speech data to obtain multiple frequency domain speech data with different resolutions, calling multiple frequency domain discrimination models with different resolutions, respectively discriminating the frequency domain speech data with the same resolution, and obtaining multiple third discrimination results, wherein the third discrimination results are used to indicate whether the target speech data is generated by calling the speech processing model; The speech discrimination module is further used to perform multiple pooling processes on the target speech data to obtain multiple time-domain speech data of different scales, call multiple time-domain discrimination models of different scales, and discriminate the time-domain speech data of the same scale respectively to obtain multiple fourth discrimination results, wherein the fourth discrimination results are used to indicate whether the target speech data is generated by calling the speech processing model; The speech discrimination module is further used to call the frequency domain discrimination model and the time domain discrimination model respectively to discriminate the sample speech data to obtain a second discrimination result, wherein the second discrimination result is used to indicate whether the sample speech data is generated by calling the speech processing model; A model training module is used to perform mel-spectrogram extraction on the sample speech data and the target speech data respectively to obtain sample mel-spectrogram features corresponding to the sample speech data and target mel-spectrogram features corresponding to the target speech data; wherein the mel-spectrogram features are features optimized for auditory perception of human ears; The model training module is used to train the speech processing model according to the difference between the sample speech data and the target speech data, the multiple third discrimination results and the multiple fourth discrimination results, the difference between the sample Mel-spectrogram feature and the target Mel-spectrogram feature, and the difference between the multiple third discrimination results and the multiple fourth discrimination results and the second discrimination result respectively.
5. The device according to claim 4, characterized in that The model training module is also used to train a speech discrimination model according to the multiple third discrimination results, the multiple fourth discrimination results and the second discrimination result.
6. The device according to claim 4, characterized in that The speech processing module is further used for: The trained speech processing model is called to encode any speech data to obtain a first speech feature, and the first speech feature is decoded to obtain speech data having a quality parameter higher than that of any speech data.
7. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the operations performed in the voice data processing method according to any one of claims 1 to 3.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the operations performed in the voice data processing method according to any one of claims 1 to 3.
