End-to-end speech recognition model training method, speech decoding method and related devices

Abstract

The present disclosure provides an end-to-end speech recognition model training method, speech decoding method, device, electronic device, and computer-readable storage medium, which relate to the fields of artificial intelligence technology such as speech recognition, natural speech processing, and deep learning. A specific implementation of the method includes: obtaining multiple sample speech files, and packaging each sample speech file into a sample file block, then generating address information of the sample file block, and then using a data loader to read the address information to generate a batch data set, and finally, training the initial end-to-end speech recognition model based on the batch data set to obtain an end-to-end speech recognition model. The application of the implementation provides an improvement in the training efficiency when training the initial end-to-end speech recognition model, and improves the quality of the end-to-end speech recognition model obtained by training.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence technologies such as speech recognition, natural speech processing, and deep learning, and in particular to a method for training an end-to-end speech recognition model and speech decoding, as well as corresponding devices, electronic devices, and computer-readable storage media. Background Art

[0002] Speech processing technology has made great strides thanks to breakthroughs in artificial intelligence and machine learning, advances in algorithms and hardware / software capabilities, and the availability of a large and diverse voice database for training multi-parameter, large-scale speech recognition and synthesis models.

[0003] With the progress of end-to-end neural networks in machine translation, speech generation, etc., end-to-end speech recognition has also achieved performance comparable to traditional methods. Unlike traditional methods that decompose speech recognition tasks into multiple subtasks (lexical model, acoustic model, and language model), end-to-end speech recognition models are usually built based on neural network temporal classification (Connectionist Temporal Classification, referred to as CTC). When using Mel spectrograms as input, they can directly generate corresponding natural language text, which greatly simplifies the model training process and is increasingly attracting attention from academia and industry. Summary of the invention

[0004] The embodiments of the present disclosure provide a method, device, electronic device, computer-readable storage medium, and computer program product for training an end-to-end speech recognition model, a speech decoding method, a device, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] In a first aspect, an embodiment of the present disclosure proposes an end-to-end speech recognition model training method, comprising: obtaining multiple sample speech files, and packaging each sample speech file into a sample file block; generating address information of the sample file block; using a data loader to read the address information and generate a batch data set; training an initial end-to-end speech recognition model based on the batch data set to obtain an end-to-end speech recognition model.

[0006] In the second aspect, an embodiment of the present disclosure proposes an end-to-end speech recognition model training device, comprising: a sample acquisition and packaging unit, including a sample acquisition subunit and a sample packaging subunit, wherein the sample acquisition subunit is configured to acquire multiple sample voice files, and the sample packaging subunit is configured to package each sample voice file into a sample file block; an address information generation unit, configured to generate address information of the sample file block; a batch data set generation unit, configured to read the address information using a data loader to generate a batch data set; a model training unit, configured to train an initial end-to-end speech recognition model based on the batch data set to obtain an end-to-end speech recognition model.

[0007] In a third aspect, an embodiment of the present disclosure proposes a speech decoding method, comprising: streaming reading of a speech file; in response to the reading time length of the speech file satisfying a preset time threshold requirement, inputting the read target speech file into an end-to-end speech recognition model for processing, and generating a decoding result corresponding to the target speech file, wherein the end-to-end speech recognition model is obtained according to the end-to-end speech recognition model training method as described in any implementation method in the first aspect.

[0008] In a fourth aspect, an embodiment of the present disclosure proposes a speech decoding device, comprising: a speech decoding unit, configured to input the read target voice file into an end-to-end speech recognition model for processing in response to the reading time length of the voice file satisfying a preset time threshold requirement, and generate a decoding result corresponding to the target voice file, wherein the end-to-end speech recognition model is obtained according to the end-to-end speech recognition model training device described in any implementation method in the second aspect.

[0009] In a fifth aspect, an embodiment of the present disclosure provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that when the at least one processor executes, it is able to implement the end-to-end speech recognition model training method described in any implementation manner in the first aspect or the speech decoding method described in any implementation manner in the third aspect.

[0010] In the sixth aspect, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, which are used to enable a computer to implement an end-to-end speech recognition model training method as described in any implementation method in the first aspect or a speech decoding method as described in any implementation method in the third aspect when executed.

[0011] In the seventh aspect, an embodiment of the present disclosure provides a computer program product comprising a computer program, which, when executed by a processor, can implement the end-to-end speech recognition model training method as described in any implementation manner in the first aspect or the speech decoding method as described in any implementation manner in the third aspect.

[0012] The end-to-end speech recognition model training and speech decoding method provided by the embodiments of the present disclosure can reduce the use of cache resources when training the initial end-to-end speech recognition model, improve the training efficiency when training the initial end-to-end speech recognition model, and make it possible to train the initial end-to-end model using large-scale sample speech files, thereby improving the quality of the trained end-to-end speech recognition model.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Other features, objects and advantages of the present disclosure will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0015] Figure 1 is an exemplary system architecture in which the present disclosure may be applied;

[0016] Figure 2 A flowchart of an end-to-end speech recognition model training method provided in an embodiment of the present disclosure;

[0017] Figure 3 A flowchart of an implementation method for obtaining a sample file block under another end-to-end speech recognition model training method provided in an embodiment of the present disclosure;

[0018] Figure 4a , 4b A flow chart of an end-to-end speech recognition model training method and a speech decoding method in a specific application scenario provided by an embodiment of the present disclosure;

[0019] Figure 5 A structural block diagram of an end-to-end speech recognition model training device provided in an embodiment of the present disclosure;

[0020] Figure 6 A structural block diagram of a speech decoding device provided in an embodiment of the present disclosure;

[0021] Figure 7 A schematic diagram of the structure of an electronic device suitable for executing an end-to-end speech recognition model training method and / or a speech decoding method provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description. It should be noted that the embodiments in the present disclosure and the features in the embodiments may be combined with each other without conflict.

[0023] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved all comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0024] Figure 1 An exemplary system architecture 100 is shown to which embodiments of the method, apparatus, electronic device, and computer-readable storage medium for training a face recognition model and recognizing a face of the present application can be applied.

[0025] like Figure 1 As shown, the system architecture 100 may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0026] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 and server 105 can be installed with various applications for realizing information communication between the two, such as speech recognition applications, online translation applications, etc.

[0027] Terminal devices 101, 102, 103 and server 105 can be hardware or software. When terminal devices 101, 102, 103 are hardware, they can be various electronic devices with display screens, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, etc.; when terminal devices 101, 102, 103 are software, they can be installed in the electronic devices listed above, which can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here. When server 105 is hardware, it can be implemented as a distributed server cluster composed of multiple servers, or as a single server; when the server is software, it can be implemented as multiple software or software modules, or as a single software or software module, and no specific limitation is made here.

[0028] The server 105 can provide various services through various built-in applications. Taking the speech recognition application that can provide speech recognition services to users as an example, the server 105 can achieve the following effects when running the speech recognition application: the server 105 obtains the voice file uploaded by the user in a streaming manner, and then the server 105 responds when the reading time length of the voice file meets the preset time threshold requirement, inputs the read target voice file into the end-to-end speech recognition model for processing, and after continuously generating the decoding results corresponding to each segment of the target voice file, summarizes the decoding results as the speech recognition result of the content in the voice file and returns it to the user.

[0029] Among them, the end-to-end speech recognition model can be trained by the end-to-end speech recognition model training application built into the server 105 according to the following steps: the server 105 obtains multiple sample voice files and packages each sample voice file into a sample file block. Then, the server 105 generates address information of the sample file block. Next, the server 105 uses a data loader to read the address information and generate a batch data set. Finally, the server 105 trains the initial end-to-end speech recognition model based on the batch data set to obtain an end-to-end speech recognition model.

[0030] Since training an end-to-end speech recognition model requires more computing resources and stronger computing power, the end-to-end speech recognition model training method provided in the subsequent embodiments of this application is generally performed by a server 105 with stronger computing power and more computing resources. Correspondingly, the end-to-end speech recognition model training device is generally also arranged in the server 105. However, it should also be pointed out that when the terminal devices 101, 102, and 103 also have computing power and computing resources that meet the requirements, the terminal devices 101, 102, and 103 can also complete the above-mentioned operations handed over to the server 105 through the training application of the end-to-end speech recognition model installed thereon, and then output the same result as the server 105. Correspondingly, the end-to-end speech recognition model training device can also be arranged in the terminal devices 101, 102, and 103. In this case, the exemplary system architecture 100 may also not include the server 105 and the network 104.

[0031] Of course, the server used to train the end-to-end speech recognition model may be different from the server that calls the trained end-to-end speech recognition model for use. In particular, the end-to-end speech recognition model trained by the server 105 can also be used to obtain a lightweight end-to-end speech recognition model suitable for being placed in the terminal devices 101, 102, and 103 by model distillation, that is, it is possible to flexibly choose to use the lightweight end-to-end speech recognition model in the terminal devices 101, 102, and 103 or the more complex end-to-end speech recognition model in the server 105 according to the recognition accuracy actually required.

[0032] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0033] Please refer to Figure 2 , Figure 2 A flowchart of an end-to-end speech recognition model training method provided in an embodiment of the present disclosure, wherein process 200 includes the following steps:

[0034] Step 201: obtain multiple sample voice files, and pack each sample voice file into a sample file block.

[0035] In this embodiment, the execution subject of the end-to-end speech recognition model training method (for example Figure 1The server 105 shown in the figure obtains multiple sample voice files, wherein the sample voice files generally include audio and text data corresponding to the audio, so as to use the audio included in the sample voice files as input and the text data corresponding to the audio as output to train the initial end-to-end speech recognition model. After the above-mentioned execution subject obtains multiple sample voice files, each sample voice file can be packaged into a sample file block, wherein the sample file block can be pre-configured, or after obtaining at least two sample voice files, the obtained sample voice files can be packaged and encapsulated.

[0036] It should be understood that when there are differences in the acquisition time, batch, etc. of the sample voice files, the sample voice files acquired later can be packaged into the sample file blocks generated based on the sample voice files acquired earlier, or new sample file blocks can be formed by packaging and encapsulating only the sample voice files acquired later.

[0037] In some embodiments, the sample file block includes at least one of the following: Tar format file, Npz format file, Hierarchical Data Format Version 5 format file, Pickle format file, among which the Tar format was originally designed to back up files to tape (Tape Archive), hence the name Tar. It can merge multiple files into one file, and the suffix of the packaged file is also "tar". The sample file block obtained in this format runs stably and can be directly run under Linux and other systems, which is fast and reliable.

[0038] Npz format files are offline datasets of handwritten digits. They can be directly imported locally without the need to manually split training data, test data, and corresponding digital labels. As a dataset for introductory deep learning projects, all digital images can be printed out in Python.

[0039] The Hierarchical Data Format Version 5 (HDF5) format file is a cross-platform data storage file that can store different types of images and digital data and can be transmitted on different types of machines. There is also a function library that uniformly handles this file format.

[0040] Pickle format file is a storage file that can save some temporary variables used in the Python project process, or strings, lists, dictionaries and other data that need to be extracted and temporarily stored.

[0041] It should be noted that the sample voice file can be obtained by the above-mentioned execution subject directly from a local storage device, or from a non-local storage device (such as Figure 1The local storage device may be a data storage module provided in the above-mentioned execution subject, such as a server hard disk, in which case the sample voice file can be quickly read locally; the non-local storage device may also be any other electronic device provided for storing data, such as some user terminals, in which case the above-mentioned execution subject may obtain the required sample voice file by sending an acquisition command to the electronic device.

[0042] Step 202: Generate address information of a sample file block.

[0043] In this embodiment, after the sample file blocks are generated based on the above step 201, each sample file block is stored respectively, and based on the storage location of each sample file block, corresponding address information is generated and recorded, so that the corresponding sample file block can be directly retrieved based on the address information later.

[0044] In practice, after the address information of each sample file block is generated, a directory file for recording the address information, such as Manifest, may also be generated, so as to facilitate reading the address information of each sample file block using the directory file.

[0045] Step 203: Use a data loader to read address information and generate a batch data set.

[0046] In this embodiment, after determining the sample file block for training, a data loader is used to obtain the address information of the sample file block, and after obtaining the corresponding sample file block based on the address information, the sample voice file included in the sample file block is read, and each sample voice file is spliced ​​into a batch data set (Batch).

[0047] In practice, when there are multiple Dataloaders and multiple sample file blocks, multiple Dataloaders are usually used to process the sample file blocks to improve reading efficiency, that is, different sample file blocks are sent to each Dataloader in parallel for reading, and the reading results of each Dataloader are aggregated to generate the Batch.

[0048] In some embodiments, the Dataloader may be configured to preload address information of a preset number of sample file blocks to avoid initial reading jams caused by low efficiency in reading address information in the initial stage.

[0049] Step 204: Train the initial end-to-end speech recognition model based on the batch data set to obtain an end-to-end speech recognition model.

[0050] In this embodiment, the batch data set obtained in the above step 203 is used, and the audio in the sample speech file included in the batch data set is continuously used as input and the corresponding text is used as output to train the initial end-to-end speech recognition model to obtain an end-to-end speech recognition model.

[0051] In practice, the initial end-to-end speech recognition model can be constructed using one of the deep speech recognition models (DeepSpeech2), Conformer, and Transformer models. Deepseech2 is a non-autoregressive model, while Conformer and Transformer are both forward reasoning autoregressive models that support non-autoregression. The above models can all support decoding in the form of prefix beam search. Among them, Transformer, as a deep learning model, has been widely verified to perform better in the field of speech models than models based on recurrent neural networks (RNNs). Conformer is a model obtained by combining Transformer with convolutional neural networks.

[0052] Among them, DeepSpeech2 can use linear spectrum graph to extract audio features, and use a 2-layer convolutional neural network as the downsampling module of the model, and then connect a multi-layer RNN layer as the encoder module (Encoder). RNN uses a unidirectional RNN structure, so it can support streaming recognition. Finally, the output of the encoder passes through the normalized exponential function layer (Softmax) to obtain the output probability, and the output probability is decoded by the decoding layer to obtain the final decoding result.

[0053] The structure of Conformer can be basically divided into two parts: Encoder and Decoder module (Decoder). During the training process, both Encoder and Decoder will have loss functions (Loss). Encoder's Loss uses the Connectionist Temporal Classification Loss of the neural network, and Decoder's Loss uses the Label Smoothing Loss. Conformer's Encoder subsampling layer, position encoder layer and multiple layers of Conformer Layer are composed of four parts, namely the Feed Forward layer, the Multi-head Attention layer, the convolution module and the Feedforward layer. Each part will use the residual connection method to avoid the gradient disappearance phenomenon. Conformer's Decoder uses the Transformer's Decoder module. It is mainly composed of a decoder input layer (OutputEmbedding), a position vector (Position Embedding) layer and multiple layers of Transformer layers. Each Transformer layer is composed of a masked multi-head attention mechanism (Masked Multi-head Attention) layer, a Multi-headAttention and a Feed Forward layer.

[0054] The end-to-end speech recognition model training method provided by the embodiments of the present disclosure can reduce the use of cache resources when training the initial end-to-end speech recognition model, improve the training efficiency when training the initial end-to-end speech recognition model, and make it possible to train the initial end-to-end model using large-scale sample speech files, thereby improving the quality of the trained end-to-end speech recognition model.

[0055] In some optional implementations of the present embodiment, the method of obtaining multiple sample voice files and packaging each sample voice file into a sample file block includes: obtaining multiple sample voice files and generating multiple sample voice file sets, wherein the sample voice file set includes a preset number of sample voice files; and packaging and processing each sample voice file set separately to generate a sample file block corresponding to each sample voice file set.

[0056] Specifically, after obtaining multiple sample voice files, each sample voice file is grouped according to a preset number and a corresponding sample voice file set is generated, wherein each sample voice file set includes a preset number of sample voice files. After the sample voice file set is generated, the sample voice file sets are packaged to form sample file blocks, so as to quickly and evenly generate sample file blocks based on the sample voice files, so as to avoid affecting the use effect of the sample file blocks due to the uneven number of sample voice files included in the sample file blocks.

[0057] In practice, in the process of generating a sample voice file set based on a sample voice file, whether the same sample voice file can be reused can be set according to actual needs, that is, whether different sample voice file sets can include the same sample voice file.

[0058] In some optional implementations of this embodiment, the method further includes: pre-configuring at least one sample file block; and adding a type list corresponding to the sample file block, wherein the type list is used to mark the type of the sample voice files existing in the corresponding sample file block.

[0059] Specifically, a sample file block can be obtained by pre-configuring or acquiring a sample voice file based on historical data, and a type list can be added to the sample file block accordingly. The type list records the types of sample voice files included in the sample file block, so that the sample file block of the same type as the training purpose can be determined through the type list, and the storage strategy of the sample voice file can be adjusted according to the type list to improve the quality of the sample file block.

[0060] In practice, the type of the sample voice file may be based on the language scene, language type, timbre type, etc. corresponding to the sample voice file.

[0061] In some optional implementations of this embodiment, it also includes: obtaining a first number of data loaders used to generate the batch data set and a second number of the sample file blocks; in response to the second number being unable to divide the first number evenly, obtaining a third number and a fourth number that can divide the first number evenly, the third number being the number closest to the second number on the left side of the second number axis, and the fourth number being the number closest to the second number on the right side of the second number axis; obtaining a first quantity difference between the third number and the second number and a second quantity difference between the fourth number and the second number; in response to the first quantity difference being less than the second quantity difference, deleting the sample file blocks equal to the second quantity difference.

[0062] Specifically, when using multiple Dataloaders to process sample file blocks, in order to ensure that each Dataloader can evenly distribute the sample file blocks, it is also possible to obtain a first number of Dataloaders and a second number of sample file blocks, and respond when the second number cannot divide the first number, obtain a third number that is closest to the left side of the second number axis and is evenly divisible by the first number, and a fourth number that is closest to the right side of the second number axis and is evenly divisible by the first number, obtain a first number difference between the third number and the second number and a second number difference between the fourth number and the second number, and respond when the first number difference is less than the second number difference, delete the sample file blocks by the second number difference, so as to adjust the existing sample file blocks by reduction, and ensure that each Dataloader can evenly distribute the sample file blocks.

[0063] In some optional implementations of this embodiment, the method further includes: in response to the first quantity difference being greater than the second quantity difference, selecting the second quantity difference basic sample file blocks from the sample file blocks, and copying the basic sample file blocks once respectively.

[0064] Specifically, when the first quantity difference is greater than the second quantity difference, a response is performed, and after randomly selecting sample file blocks with the second quantity difference from the existing sample quantity blocks as basic sample file blocks, a single copy is performed on each basic sample file block, so as to adjust the existing sample file blocks by filling the sample file blocks, thereby ensuring that each Dataloader can evenly distribute the sample file blocks.

[0065] Please refer to Figure 3 , Figure 3 In an end-to-end speech recognition model training method provided by an embodiment of the present disclosure, in the case where a sample file block and a type list corresponding to the sample file block are pre-configured, Figure 2 Step 201 in the process 200 shown provides a specific implementation process 300, and the other steps in the process 200 are not adjusted. The specific implementation provided by this embodiment is replaced by step 201 to obtain a new complete embodiment. The process 300 includes the following steps:

[0066] Step 301: Acquire multiple sample voice files and a first type of each sample voice file.

[0067] In this embodiment, while acquiring a plurality of sample voice files, the first type of each sample voice file is acquired.

[0068] Step 302: Determine the second type missing from the sample file block based on the type list.

[0069] In this embodiment, the second type missing in each sample file block is determined based on the type list of each sample file block, that is, the sample voice file of the second type does not exist in the current sample file block.

[0070] Step 303: In response to the first type of the sample voice file matching the second type, packing the sample voice file into a sample file block.

[0071] In this embodiment, when the first type of the sample voice file matches the second type missing in the sample file block, a response is made and the sample voice file is correspondingly stored in the sample file block.

[0072] This embodiment Figure 2 On the basis of the illustrated embodiment, the types of sample voice files missing from each sample file block can be further determined based on the type list, and the sample voice files of the types missing from the sample file block can be stored accordingly, so as to expand the types of sample voice files stored in the sample file block and enhance the use value of the sample file block.

[0073] In practice, if it is desired to use sample voice files of the same type to train the initial end-to-end speech recognition model, then based on the similarity between the first type of the sample voice file and the type of the sample voice files already in the sample file block recorded in the type list, the sample voice files whose similarity meets the preset threshold requirement can be determined and stored.

[0074] On the basis of any of the above embodiments, it also includes: obtaining an intermediate decoding result in the initial end-to-end speech recognition model training process; using a scene semantic model to process the intermediate decoding result to generate a score for the intermediate decoding result, wherein the scene semantic model is used to generate a score for the semantic quality of the input content in a language scenario corresponding to the scene semantic model; in response to the sum of the scores of the intermediate decoding results exceeding a preset score threshold, adding an available scene label for the language scenario corresponding to the scene semantic model to the end-to-end speech recognition model.

[0075] Specifically, in the process of training the initial end-to-end speech recognition model, taking a single training as an example, an intermediate decoding result generated inside the initial end-to-end speech recognition model in the process of taking the audio in the sample speech file as input and the corresponding text data as output is obtained, and then the intermediate result is processed using a scene semantic model for generating a score of the semantic quality of the input content in the language scenario corresponding to the scene semantic model to generate a score corresponding to the intermediate decoding result, and after completing the training of the initial end-to-end speech recognition model and obtaining the end-to-end speech recognition model, the scores of the intermediate decoding results are added to obtain a score and value, and when the score value and the sum exceed the score threshold requirement, a response is given to add an available scene label corresponding to the scene semantic model to the end-to-end speech recognition model, so as to indicate the language scenario available to the end-to-end speech recognition model through the available scene label.

[0076] The above embodiments describe how to train an end-to-end speech recognition model from various aspects. In order to highlight the effect of the trained end-to-end speech recognition model from the actual usage scenario as much as possible, the present disclosure also specifically provides a solution for using the trained end-to-end speech recognition model to solve practical problems. A speech decoding method includes the following steps:

[0077] The voice file is read in a streaming manner, and a response is made when the reading time length of the voice file meets the preset time threshold requirement. The read target voice file is input into the end-to-end voice recognition model for processing, and a decoding result corresponding to the target voice file is generated. The end-to-end voice recognition model can be trained based on the end-to-end voice recognition model training method in any of the above embodiments.

[0078] Among them, in the process of streaming reading of voice files and generating target voice files, if there are multiple target voice files, the starting point of the subsequent target voice file can be the end point of the adjacent previous target voice file, or it can be preferably a point a preset time before the end point of the previous target voice file as the starting point, so as to avoid the recognition quality of the connection position of the two target voice files caused by reasons such as streaming reading delay.

[0079] To deepen understanding, the present disclosure also provides a specific implementation solution in combination with a specific application scenario:

[0080] like Figure 4a As shown, multiple sample voice files (sample voice file 1, sample voice file 2, sample voice file 3, ...) are obtained respectively, and each sample voice file is packaged into a sample file block (Tar1, Tar2, Tar3, ...), and after the address information of each sample file block is generated, the location of each sample file block is recorded using Manifest.

[0081] Then, Dataloader After using Manifest to find the location of each sample file block, a batch dataset is generated to train the initial end-to-end speech recognition model built on DeepSpeech2.

[0082] After completing the training and obtaining the end-to-end speech recognition model, the voice file is streamed and read. When the reading time length of the voice file meets the preset time threshold requirement, a response is made, and the read voice file is generated and recorded as the target voice file 1. The preset time length from the end of the target voice file 1 is taken as the starting point, and the voice file is read to reach the target voice file 2, the target voice file 3, etc. with the same time length as the target voice file 1 in turn.

[0083] The target voice files 1, 2, 3, ... are sequentially input into the end-to-end voice recognition model, and the end-to-end voice recognition model is used to process the target voice files 1, 2, ... sequentially, such as Figure 4b , a feature extraction layer built based on a linear spectrogram is used to extract features from the original audio in the voice file, and then a 2-layer convolutional neural network is used as the downsampling module of the model, and then a multi-layer RNN is connected as the Encoder. The RNN uses a unidirectional RNN structure to support streaming recognition.

[0084] Finally, the output of the encoder is passed through the Softmax layer to obtain the output probability, and then decoded by the decoding layer to obtain the final decoding result.

[0085] Further references Figure 5 and Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides an end-to-end speech recognition model training device embodiment and a speech decoding device embodiment, the end-to-end speech recognition model training device embodiment and Figure 2 The end-to-end speech recognition model training method embodiment shown corresponds to the speech decoding device embodiment and the speech decoding method embodiment. The above device can be specifically applied to various electronic devices.

[0086] like Figure 5As shown, the end-to-end speech recognition model training device 500 of this embodiment may include: a sample acquisition and packaging unit 501, an address information generation unit 502, a batch data set generation unit 503 and a model training unit 504. The sample acquisition and packaging unit 501 includes a sample acquisition subunit and a sample packaging subunit, wherein the sample acquisition subunit is configured to acquire a plurality of sample speech files, and the sample packaging subunit is configured to package each sample speech file into a sample file block; the address information generation unit 502 is configured to generate the address information of the sample file block; the batch data set generation unit 503 is configured to read the address information using a data loader to generate a batch data set; the model training unit 504 is configured to train the initial end-to-end speech recognition model based on the batch data set to obtain an end-to-end speech recognition model.

[0087] In this embodiment, in the end-to-end speech recognition model training device 500, the specific processing of the sample acquisition and packaging unit 501, the address information generation unit 502, the batch data set generation unit 503 and the model training unit 504 and the technical effects thereof can be referred to respectively. Figure 2 The relevant descriptions of steps 201 - 204 in the corresponding embodiment are not repeated here.

[0088] In some optional implementations of the present embodiment, the sample acquisition and packaging unit 501 includes: the sample acquisition subunit is further configured to acquire multiple sample voice files and generate multiple sample voice file sets, wherein the sample voice file set includes a preset number of sample voice files; the sample packaging subunit is further configured to package and process each sample voice file set separately to generate a sample file block corresponding to each sample voice file set.

[0089] In some optional implementations of the present embodiment, the end-to-end speech recognition model training device 500 may further include: a file block configuration unit, configured to pre-configure at least one of the sample file blocks; a type list adding unit, configured to add a type list corresponding to the sample file block, wherein the type list is used to mark the type of the sample speech file already in the corresponding sample file block.

[0090] In some optional implementations of the present embodiment, the sample acquisition and packaging unit 501 includes: the sample acquisition subunit is further configured to acquire multiple sample voice files and the first type of each sample voice file; the sample packaging subunit is further configured to determine the second type missing in the sample file block based on the type list; in response to the first type of the sample voice file matching the second type, packaging the sample voice file into the sample file block.

[0091] In some optional implementations of the present embodiment, the end-to-end speech recognition model training device 500 may also include: an intermediate result acquisition unit, configured to obtain the intermediate decoding result in the initial end-to-end speech recognition model training process; an intermediate result scoring unit, configured to process the intermediate decoding result using a scene semantic model to generate a score for the intermediate decoding result, wherein the scene semantic model is used to generate a score for the semantic quality of the input content in a language scenario corresponding to the scene semantic model; a scene label adding unit, configured to add an available scene label for the language scenario corresponding to the scene semantic model to the end-to-end speech recognition model in response to the score and value obtained by adding the scores of the intermediate decoding results exceeding a preset score threshold.

[0092] In some optional implementations of the present embodiment, the end-to-end speech recognition model training device 500 may further include: a loader quantity and file block quantity acquisition unit, including a loader quantity acquisition subunit and a file block quantity acquisition subunit, wherein the loader quantity acquisition subunit is configured to acquire a first quantity of data loaders used to generate the batch data set, and the file block quantity acquisition subunit is configured to acquire a second quantity of the sample file blocks; a divisible quantity acquisition unit, configured to, in response to the second quantity being unable to divide the first quantity, acquire a third quantity and a fourth quantity that can divide the first quantity, wherein the third quantity is the number closest to the second quantity on the left side of the second quantity axis, and the fourth quantity is the number closest to the second quantity on the right side of the second quantity axis; a quantity difference acquisition unit, configured to acquire a first quantity difference between the third quantity and the second quantity and a second quantity difference between the fourth quantity and the second quantity; a first sample file block adjustment unit, configured to, in response to the first quantity difference being less than the second quantity difference, delete the second quantity difference number of the sample file blocks.

[0093] In some optional implementations of this embodiment, the end-to-end speech recognition model training device 500 may also include: a second sample file block adjustment unit, configured to select the second number difference basic sample file blocks from the sample file block in response to the first number difference being greater than the second number difference, and to copy the basic sample file blocks once respectively.

[0094] In some optional implementations of this embodiment, the sample file block includes at least one of the following: a Tar format file, an Npz format file, a Hierarchical Data Format Version 5 format file, and a Pickle format file.

[0095] like Figure 6As shown, the speech decoding device 600 of this embodiment may include: a speech file streaming reading unit 601 and a speech decoding unit 602. The speech file streaming reading unit 601 is configured to stream read a speech file; the speech decoding unit 602 is configured to input the read target speech file into the end-to-end speech recognition model for processing in response to the reading time length of the speech file meeting the preset time threshold requirement, and generate a decoding result corresponding to the target speech file; wherein the end-to-end speech recognition model is obtained according to the end-to-end speech recognition model training device 500.

[0096] In this embodiment, in the speech decoding device 600, the specific processing of the speech file streaming reading unit 601 and the speech decoding unit 602 and the technical effects brought about by them can respectively correspond to the relevant descriptions in the method embodiment, and will not be repeated here.

[0097] This embodiment exists as an apparatus embodiment corresponding to the above-mentioned method embodiment. The end-to-end speech recognition model training device and speech decoding device provided by this embodiment can reduce the use of cache resources when training the initial end-to-end speech recognition model, and can improve the training efficiency when training the initial end-to-end speech recognition model, making it possible to train the initial end-to-end model using large-scale sample speech files, thereby improving the quality of the trained end-to-end speech recognition model.

[0098] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can implement the end-to-end speech recognition model training method and / or speech decoding method described in any of the above embodiments when executing.

[0099] According to an embodiment of the present disclosure, the present disclosure also provides a readable storage medium, which stores computer instructions, and the computer instructions are used to enable a computer to implement the end-to-end speech recognition model training method and / or speech decoding method described in any of the above embodiments when executed.

[0100] An embodiment of the present disclosure provides a computer program product, which, when executed by a processor, can implement the end-to-end speech recognition model training method and / or speech decoding method described in any of the above embodiments.

[0101] Figure 7A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0102] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0103] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0104] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as an end-to-end speech recognition model training method and / or a speech decoding method. For example, in some embodiments, the end-to-end speech recognition model training method and / or the speech decoding method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the end-to-end speech recognition model training method and / or the speech decoding method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the end-to-end speech recognition model training method and / or the speech decoding method in any other suitable manner (e.g., by means of firmware).

[0105] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0106] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0107] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0108] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0109] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0110] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and virtual private servers (VPS) services.

[0111] The technical solution of the disclosed embodiment can reduce the use of cache resources when training the initial end-to-end speech recognition model, improve the training efficiency when training the initial end-to-end speech recognition model, and make it possible to train the initial end-to-end model using large-scale sample speech files, thereby improving the quality of the trained end-to-end speech recognition model.

[0112] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0113] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An end-to-end speech recognition model training method, comprising: Obtaining a first number of data loaders for generating a batch data set and a second number of sample file chunks; In response to the second number not being able to divide the first number, obtaining a third number and a fourth number that can divide the first number, wherein the third number is the number closest to the second number on the left side of the number axis of the second number, and the fourth number is the number closest to the second number on the right side of the number axis of the second number; obtaining a first number difference between the third number and the second number and a second number difference between the fourth number and the second number; in response to the first number difference being less than the second number difference, deleting the sample file blocks equal to the second number difference; Acquiring a plurality of sample voice files and packaging each of the sample voice files into the sample file block, including: acquiring a plurality of sample voice files and generating a plurality of sample voice file sets, wherein the sample voice file sets include a preset number of sample voice files; packaging and processing each of the sample voice file sets respectively to generate a sample file block corresponding to each of the sample voice file sets; Generate address information of the sample file block, wherein the address information is recorded using Manifest; Using the data loader to read the address information by reading the Manifest, and generating the batch data set; Training an initial end-to-end speech recognition model based on the batch data set to obtain an end-to-end speech recognition model; It also includes: obtaining intermediate decoding results in the initial end-to-end speech recognition model training process; using a scene semantic model to process the intermediate decoding results to generate a score for the intermediate decoding results, wherein the scene semantic model is used to generate a score for the semantic quality of the input content in a language scenario corresponding to the scene semantic model; in response to the score and value obtained by adding the scores of each of the intermediate decoding results exceeding a preset score threshold, adding an available scene label for the language scenario corresponding to the scene semantic model to the end-to-end speech recognition model.

2. The method according to claim 1, further comprising: pre-configuring at least one of the sample file blocks; A type list is added corresponding to the sample file block, wherein the type list is used to mark the types of sample voice files existing in the corresponding sample file block.

3. The method according to claim 2, wherein: The step of obtaining a plurality of sample voice files and packaging each of the sample voice files into a sample file block includes: Acquire a plurality of sample voice files and a first type of each of the sample voice files; Determining a second type missing from the sample file block based on the type list; In response to the first type of the sample voice file matching the second type, the sample voice file is packaged into a sample file block.

4. The method according to claim 1, further comprising: In response to the first quantity difference being greater than the second quantity difference, selecting the second quantity difference basic sample file blocks from the sample file blocks, and respectively copying the basic sample file blocks once.

5. The method according to any one of claims 1 to 4, wherein: The sample file block includes at least one of the following: a Tar format file, an Npz format file, a Hierarchical Data Format Version 5 format file, and a Pickle format file.

6. A speech decoding method, comprising: Streaming reading of audio files; In response to the reading time length of the voice file meeting a preset time threshold requirement, the read target voice file is input into an end-to-end voice recognition model for processing to generate a decoding result corresponding to the target voice file, wherein the end-to-end voice recognition model is obtained based on the end-to-end voice recognition model training method described in any one of claims 1-5.

7. An end-to-end speech recognition model training device, comprising: A loader quantity and file block quantity acquisition unit, comprising a loader quantity acquisition subunit and a file block quantity acquisition subunit, wherein the loader quantity acquisition subunit is configured to acquire a first quantity of data loaders used to generate a batch data set, and the file block quantity acquisition subunit is configured to acquire a second quantity of sample file blocks; a divisible quantity acquisition unit, configured to, in response to the second quantity being unable to divide the first quantity, acquire a third quantity and a fourth quantity that can divide the first quantity, wherein the third quantity is the number closest to the second quantity on the left side of the second quantity axis, and the fourth quantity is the number closest to the second quantity on the right side of the second quantity axis; a quantity difference acquisition unit, configured to acquire a first quantity difference between the third quantity and the second quantity, and a second quantity difference between the fourth quantity and the second quantity; a first sample file block adjustment unit, configured to, in response to the first quantity difference being smaller than the second quantity difference, delete the second quantity difference number of sample file blocks; A sample acquisition and packaging unit, comprising a sample acquisition subunit and a sample packaging subunit, wherein the sample acquisition subunit is configured to acquire a plurality of sample voice files, and the sample packaging subunit is configured to package each of the sample voice files into the sample file block, and the sample acquisition subunit is further configured to acquire a plurality of sample voice files and generate a plurality of sample voice file sets, wherein the sample voice file sets include a preset number of sample voice files; the sample packaging subunit is further configured to package and process each of the sample voice file sets respectively, and generate a sample file block corresponding to each of the sample voice file sets; An address information generating unit, configured to generate address information of the sample file block, wherein the address information is recorded using Manifest; a batch data set generating unit, configured to use the data loader to read the address information by reading the Manifest, and generate the batch data set; A model training unit is configured to train an initial end-to-end speech recognition model based on the batch data set to obtain an end-to-end speech recognition model; Also includes: An intermediate result acquisition unit is configured to acquire intermediate decoding results in the process of training the initial end-to-end speech recognition model; an intermediate result scoring unit is configured to process the intermediate decoding results using a scene semantic model to generate a score for the intermediate decoding results, wherein the scene semantic model is used to generate a score for the semantic quality of the input content in a language scenario corresponding to the scene semantic model; a scene label adding unit is configured to add an available scene label for the language scenario corresponding to the scene semantic model to the end-to-end speech recognition model in response to the score and value obtained by summing up the scores of the intermediate decoding results exceeding a preset score threshold.

8. The apparatus according to claim 7, further comprising: A file block configuration unit, configured to pre-configure at least one of the sample file blocks; The type list adding unit is configured to add a type list corresponding to the sample file block, wherein the type list is used to mark the type of the sample voice files existing in the corresponding sample file block.

9. The device according to claim 8, wherein: The sample acquisition and packaging unit includes: The sample acquisition subunit is further configured to acquire a plurality of sample voice files and a first type of each of the sample voice files; The sample packing subunit is further configured to determine the second type missing from the sample file block based on the type list; In response to the first type of the sample voice file matching the second type, the sample voice file is packaged into a sample file block.

10. The apparatus according to claim 7, further comprising: The second sample file block adjustment unit is configured to select the second-number-difference basic sample file blocks from the sample file blocks in response to the first number difference being greater than the second number difference, and to copy the basic sample file blocks once respectively.

11. The device according to any one of claims 7 to 10, wherein: The sample file block includes at least one of the following: a Tar format file, an Npz format file, a Hierarchical Data Format Version 5 format file, and a Pickle format file.

12. A speech decoding device, comprising: A voice file streaming reading unit, configured to read a voice file in a streaming manner; The speech decoding unit is configured to input the read target speech file into the end-to-end speech recognition model for processing in response to the reading time length of the speech file meeting the preset time threshold requirement, and generate a decoding result corresponding to the target speech file, wherein the end-to-end speech recognition model is obtained based on the end-to-end speech recognition model training device described in any one of claims 7-11.

13. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the end-to-end speech recognition model training method described in any one of claims 1-5 and / or the speech decoding method described in claim 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute the end-to-end speech recognition model training method described in any one of claims 1 to 5 and / or the speech decoding method described in claim 6.

Images