Multi-dialect speech recognition method and device based on synthetic speech data

Abstract

The embodiment of the invention provides a multi-dialect speech recognition method and device based on synthetic speech data, equipment and a computer readable storage medium. The method comprises the following steps: acquiring initial voice data; inputting the voice data into a trained voice recognition model to generate synthetic voice data; wherein the speech recognition model can be obtained by the following steps: pre-training sample data through a preset framework to obtain a preliminary speech recognition model; and adjusting the preliminary speech recognition model based on a preset speech synthesis system, and constructing the speech recognition model. In this way, the problem that an existing data set is insufficient in quality is solved, and the problem that multi-dialect high-precision annotation data is scarce is effectively solved; the performance of the speech recognition system in a multi-dialect environment is remarkably improved, the requirements of intelligent equipment and services for high-precision speech recognition are met, and the user experience is greatly improved.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of data processing, and in particular, to a method, apparatus, device, and computer-readable storage medium for multi-dialect speech recognition based on synthetic speech data. Background Art

[0002] With the rapid development of artificial intelligence technology, speech recognition, as one of the important ways of human-computer interaction, has been widely used in many fields such as smart phones, smart homes, and smart customer service. However, the existing speech recognition system still has the problem of insufficient recognition accuracy when facing diverse voice inputs, especially voices containing multiple dialects. The main reason is that the data sets that the speech recognition model relies on during the training process often lack sufficient dialect coverage and high-quality annotations, resulting in poor performance of the model when processing dialect voices. Summary of the invention

[0003] According to an embodiment of the present application, a multi-dialect speech recognition solution based on synthetic speech data is provided, which can learn rich speech feature representations from unlabeled speech data, providing a strong foundation for speech recognition. By making careful adjustments for specific tasks and scenarios, the model can adapt to diverse dialects, accents and noise environments, greatly improving recognition accuracy and robustness.

[0004] In a first aspect of the present application, a multi-dialect speech recognition method based on synthetic speech data is provided. The method comprises: Obtaining initial voice data; Inputting the speech data into a trained speech recognition model to generate synthesized speech data; The speech recognition model can be obtained in the following way: The sample data is pre-trained through a preset framework to obtain a preliminary speech recognition model; the loss function of the preset framework includes: ; Wherein, ct is a context representation; Quantize the target vector for the current time step; is the temperature parameter; Represents the similarity measure between two vectors; The preliminary speech recognition model is adjusted based on a preset speech synthesis system to construct the speech recognition model.

[0005] In a possible implementation, the preset framework includes a Fairseq framework; The sample data includes the KeSpeech dataset.

[0006] In a possible implementation, the adjusting the preliminary speech recognition model based on a preset speech synthesis system includes: Performing phoneme alignment on the data in the KeSpeech data set to obtain target data; Extracting feature data from the target data; the feature data includes Mel spectrum, pitch and / or energy; By using a preset diffusion model, forward noise addition and reverse generation processing are performed on the characteristic data to obtain target sample data; The target sample data is used as sample data for training the preliminary speech recognition model.

[0007] In a possible implementation, the adding forward noise to the feature data by using a preset diffusion model includes: ; Among them, the is the feature data of the speech at step t; is the scaling factor for each time step; is standard Gaussian noise.

[0008] In a possible implementation, the reverse generation of the characteristic data by using a preset diffusion model includes: ; Among them, the is the feature data of the speech at step t; is the scaling factor for each time step; is standard Gaussian noise.

[0009] In a possible implementation, the method further includes: receiving feedback data from a user in response to the synthesized speech data; The speech recognition model is modified based on the feedback data.

[0010] In a possible implementation, the method further includes: The initial speech data is normalized and / or denoised.

[0011] In a second aspect of the present application, a multi-dialect speech recognition device based on synthesized speech data is provided. The device comprises: An acquisition module, used to acquire initial voice data; A generation module, used for inputting the speech data into a trained speech recognition model to generate synthesized speech data; The speech recognition model can be obtained in the following way: The sample data is pre-trained through a preset framework to obtain a preliminary speech recognition model; the loss function of the preset framework includes: ; Wherein, ct is a context representation; Quantize the target vector for the current time step; is the temperature parameter; Represents the similarity measure between two vectors; The preliminary speech recognition model is adjusted based on a preset speech synthesis system to construct the speech recognition model.

[0012] In a third aspect of the present application, an electronic device is provided, which includes a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the program, the method described above is implemented.

[0013] In a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method according to the first aspect of the present application is implemented.

[0014] The multi-dialect speech recognition method based on synthetic speech data provided in the embodiment of the present application obtains initial speech data; inputs the speech data into a trained speech recognition model to generate synthetic speech data; The speech recognition model can be obtained in the following way: The sample data is pre-trained through a preset framework to obtain a preliminary speech recognition model; the loss function of the preset framework includes: ; Wherein, ct is a context representation; Quantize the target vector for the current time step; is the temperature parameter; Represents a similarity measure between two vectors; based on a preset speech synthesis system, the preliminary speech recognition model is adjusted to construct the speech recognition model, which greatly improves the recognition accuracy and robustness.

[0015] It should be understood that the contents described in the Summary of the Invention are not intended to limit the key or important features of the embodiments of the present application, nor are they intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and other features, advantages and aspects of the embodiments of the present application will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein: Figure 1 is a flow chart of a multi-dialect speech recognition method based on synthetic speech data according to an embodiment of the present application; Figure 2 is a block diagram of a multi-dialect speech recognition device based on synthetic speech data according to an embodiment of the present application; Figure 3 It is a schematic diagram of the structure of a terminal device or a server suitable for implementing the embodiments of the present application. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0018] In addition, the term "and / or" in this article is only a description of the association relationship between the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0019] Figure 1 A flow chart of a multi-dialect speech recognition method based on synthesized speech data according to an embodiment of the present disclosure is shown. The method comprises: S110, obtaining initial voice data.

[0020] In this embodiment, the execution subject (eg, a server) of the multi-dialect speech recognition method based on synthesized speech data may obtain the initial speech data through a wired or wireless connection.

[0021] Furthermore, the execution subject may obtain language data sent by an electronic device (such as a mobile phone, a tablet computer, etc.) that is in communication with the execution subject, or may obtain voice data pre-stored locally.

[0022] Wherein, the voice data includes Mandarin and / or local dialect.

[0023] S120, inputting the speech data into a trained speech recognition model to generate synthesized speech data.

[0024] In some embodiments, the "KeSpeech dataset" is selected as the initial training sample set. This dataset is a huge corpus covering 34 cities in China, 27,237 speakers, and 1,542 hours of speech signals, including standard Mandarin and other 8 dialects. The KeSpeech dataset has data diversity and scale, which helps to improve the generalization ability of the model in different dialects and contexts.

[0025] In some embodiments, in order to further improve the accuracy of model training, the voice data obtained in step S110 may be processed as follows: Noise removal: Environmental noise can be removed through corresponding multiple filtering technologies to ensure the clarity of speech signals; Data normalization: Perform sampling rate adjustment, volume equalization and other processing on audio data to ensure that the data meets the input requirements of the model; specific standard specifications can be pre-set according to the actual application scenario.

[0026] In some embodiments, the sample data can be pre-trained through the Fairseq framework to obtain a preliminary speech recognition model; the use of the Fairseq framework for pre-training can fully tap the potential information in the KeSpeech data set and improve the basic performance of the speech recognition model. The loss function used is as follows: ; Wherein, ct is a context representation; Quantize the target vector for the current time step; is the temperature parameter; Represents the similarity measure between two vectors; Self-supervised learning through large-scale speech data can effectively capture speech features and provide a general speech recognition foundation for the subsequent fine-tuning stage, thereby reducing dependence on specific task data.

[0027] In some embodiments, the diversity and quality of data in the speech recognition process directly affect the performance of the model. In order to enhance the diversity of the data set, a self-developed synthetic speech system is used in the present disclosure to generate high-quality synthetic speech data. The system can generate speech in parallel and introduce variable-length predictors for pitch, energy, and duration, making the generated speech more natural and controllable.

[0028] Specifically, firstly, the speech data, such as the data in the KeSpeech dataset, is subjected to phoneme alignment (MFA), and then the alignment result is subjected to feature extraction and preprocessing. The extracted features include Mel-spectrogram, pitch and / or energy data to provide input for the diffusion model.

[0029] The generation process of the scattered model can be divided into two stages, namely, the forward noise addition stage and the reverse generation stage: In the forward noise addition stage, noise can be gradually added to the clean speech features, making it gradually become pure noise; In the reverse generation stage, the noise is gradually removed through diffusion model learning, and high-quality speech features are restored from pure noise.

[0030] Among them, the forward noise addition includes: ; Among them, the is the feature data of the speech at step t; is the scaling factor for each time step; is standard Gaussian noise; Reverse generation includes: ; Among them, the is the feature data of the speech at step t; is the scaling factor for each time step; is standard Gaussian noise.

[0031] Furthermore, through the above processing, noise can be effectively removed and natural, high-quality speech signals can be generated. The flexibility of the generation process and the ability of high-quality synthesis can effectively make up for the lack of real speech data and generate realistic synthetic speech data. The synthetic speech data generated by the diffusion model not only improves the naturalness of speech synthesis, but also provides more diverse training data for the model, further enhancing the generalization ability of the speech recognition model.

[0032] In some embodiments, the preliminary speech recognition model is adjusted based on a preset speech synthesis system to construct a speech recognition model, that is, the pre-trained preliminary speech recognition model is optimized using a mixed data set of synthetic and real speech data to construct a speech recognition model.

[0033] The preliminary speech recognition model obtained by pre-training is used as the basic input, and a large amount of high-quality synthetic speech data and a small amount of accurately annotated data are used for optimization: the generation of synthetic speech data provides a large number of high-quality samples, and the accurately annotated data can further refine the adaptability of the model. The small amount of accurately annotated data is pre-annotated according to the application scenario, for example, the annotated data corresponding to the same meaning of Mandarin and dialect, etc.

[0034] Furthermore, it also includes: Receive feedback data from a user in response to the synthesized speech data; and modify the speech recognition model based on the feedback data.

[0035] Specifically, the user's input language data is processed by the speech recognition model to generate a speech recognition result (synthesized speech data). The user can interactively modify the recognition result, and the modified data will generate a new sample together with the user's original input. The new sample will be used to further optimize the speech recognition model and the self-developed synthetic speech system, thus forming a closed-loop improvement mechanism. The ultimate goal of the entire process is to enable the speech recognition model to achieve more accurate and robust speech-to-text conversion in practical applications through a cycle of user feedback and model optimization.

[0036] A specific embodiment based on this application is given below: Pre-training was performed on the KeSpeech dataset, and fine-tuning was performed on the Cantonese dataset. The data used for fine-tuning was the synthetic speech generated by the speech recognition model disclosed in this disclosure, and the text content selected was "Les Miserables", "Thunderstorm", "Romance of the Three Kingdoms", "Journey to the West" and text randomly generated by GPT. In terms of data set division, two different proportion experiments were conducted: In terms of data set division, two different ratio experiments were conducted: The first experiment: The training set and the test set are divided in a 1:1 ratio, that is, 10 training data and 10 test data.

[0037] Second experiment: The training set and test set are divided in a ratio of 4:1, that is, 80 training data and 20 test data.

[0038] The average duration of each voice data is about 20 seconds. Experimental results show that under the condition of accurate annotation, using only 80 data (a total of about 33 minutes of voice duration) and running 1600 epochs on the V100 server can significantly improve the performance of the model, which is significantly improved compared to the basic model. This shows that a small amount of high-quality data combined with an effective fine-tuning strategy can achieve excellent results.

[0039] The specific experimental results are shown in Table 1: Table 1 Model Name WER (Word Error Rate) CER (Word Error Rate) Processing speed (token / s) Base_KeSpeech 51.84% 30.97% 433.49 finetuning _10 67.28% 80.95% 376.73 finetuning _100 12.60% 12.31% 543.30 The experimental results show that the word error rate and character error rate of the fine-tuned model have been greatly reduced, especially the finetuning _100 model has the best performance. This shows that by fine-tuning the model based on a small amount of high-quality synthetic speech data, the recognition performance of the model can be significantly improved. At the same time, the processing speed of the model has also been significantly improved.

[0040] According to the embodiments of the present disclosure, the following technical effects are achieved: This paper not only overcomes the problem of insufficient quality of existing data sets, but also effectively solves the problem of scarce high-precision annotated data in multiple dialects by pre-training on the Fairseq framework and using a self-developed synthetic speech system to generate high-quality professional dialect synthetic speech data for fine-tuning. The proposed method significantly improves the performance of the speech recognition system in a multi-dialect environment, meets the needs of smart devices and services for high-precision speech recognition, and greatly improves the user experience.

[0041] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.

[0042] The above is an introduction to the method embodiment. The following is a further explanation of the scheme described in this application through an apparatus embodiment.

[0043] Figure 2 FIG. 4 shows a block diagram of a multi-dialect speech recognition device based on synthesized speech data according to an embodiment of the present application, as shown in FIG. Figure 2 Shown include: An acquisition module 210, used to acquire initial voice data; A generating module 220, configured to input the speech data into a trained speech recognition model to generate synthesized speech data; The speech recognition model can be obtained in the following way: The sample data is pre-trained through a preset framework to obtain a preliminary speech recognition model; the loss function of the preset framework includes: ; Wherein, ct is a context representation; Quantize the target vector for the current time step; is the temperature parameter; Represents the similarity measure between two vectors; The preliminary speech recognition model is adjusted based on a preset speech synthesis system to construct the speech recognition model.

[0044] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0045] Figure 3 A schematic diagram of the structure of a terminal device or server suitable for implementing an embodiment of the present application is shown.

[0046] like Figure 3 As shown, the terminal device or server includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage part 308 into the random access memory (RAM) 303. In RAM 303, various programs and data required for the operation of the terminal device or server are also stored. CPU 301, ROM 302 and RAM 303 are connected to each other through a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0047] The following components are connected to the I / O interface 305: an input section 306 including a keyboard, a mouse, etc.; an output section 307 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed, so that a computer program read therefrom is installed into the storage section 308 as needed.

[0048] In particular, according to an embodiment of the present application, the above method flow steps can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 309, and / or installed from the removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, the above-mentioned functions defined in the system of the present application are executed.

[0049] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0050] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of the code, and the aforementioned module, program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0051] The units or modules involved in the embodiments described in the present application may be implemented by software or hardware. The units or modules described may also be arranged in a processor. The names of these units or modules do not, in some cases, constitute limitations on the units or modules themselves.

[0052] As another aspect, the present application further provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable storage medium stores one or more programs, and when the above programs are used by one or more processors to execute the method described in the present application.

[0053] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of application involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the aforementioned application concept. For example, the above features are replaced with (but not limited to) technical features with similar functions applied in the present application.

Claims

1. A multi-dialect speech recognition method based on synthetic speech data, characterized in that: include: Obtaining initial voice data; Inputting the speech data into a trained speech recognition model to generate synthesized speech data; The speech recognition model can be obtained in the following way: The sample data is pre-trained through a preset framework to obtain a preliminary speech recognition model; the loss function of the preset framework includes: ； Wherein, ct is a context representation; Quantize the target vector for the current time step; is the temperature parameter; Represents the similarity measure between two vectors; The preliminary speech recognition model is adjusted based on a preset speech synthesis system to construct the speech recognition model.

2. The method according to claim 1, characterized in that The preset framework includes a Fairseq framework; The sample data includes the KeSpeech dataset.

3. The method according to claim 2, characterized in that The adjusting the preliminary speech recognition model based on the preset speech synthesis system includes: Performing phoneme alignment on the data in the KeSpeech data set to obtain target data; Extracting feature data from the target data; the feature data includes Mel spectrum, pitch and / or energy; By using a preset diffusion model, forward noise addition and reverse generation processing are performed on the characteristic data to obtain target sample data; The target sample data is used as sample data for training the preliminary speech recognition model.

4. The method according to claim 3, characterized in that The adding of forward noise to the characteristic data by using a preset diffusion model includes: ； Among them, the is the feature data of the speech at step t; is the scaling factor for each time step; is standard Gaussian noise.

5. The method according to claim 4, characterized in that The reverse generation of the characteristic data by using a preset diffusion model includes: ； Among them, the is the feature data of the speech at step t; is the scaling factor for each time step; is standard Gaussian noise.

6. The method according to claim 5, characterized in that Also includes: receiving feedback data from a user in response to the synthesized speech data; The speech recognition model is modified based on the feedback data.

7. The method according to claim 1, characterized in that Also includes: The initial speech data is normalized and / or denoised.

8. A multi-dialect speech recognition device based on synthetic speech data, characterized in that: include: An acquisition module, used to acquire initial voice data; A generation module, used for inputting the speech data into a trained speech recognition model to generate synthesized speech data; The speech recognition model can be obtained in the following way: The sample data is pre-trained through a preset framework to obtain a preliminary speech recognition model; the loss function of the preset framework includes: ； Wherein, ct is a context representation; Quantize the target vector for the current time step; is the temperature parameter; Represents the similarity measure between two vectors; The preliminary speech recognition model is adjusted based on a preset speech synthesis system to construct the speech recognition model.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images