A Chinese speech recognition method, system, storage medium and electronic device

Abstract

The present application relates to a kind of Chinese speech recognition method, system, storage medium and electronic equipment, comprising: based on multiple Chinese speech training samples, the original CTC coding network added with a fine-grained loss module and two intermediate layer loss modules is trained, obtains the first Chinese speech recognition model, and delete the fine-grained loss module and the two intermediate layer loss modules in the first Chinese speech recognition model, obtain target Chinese speech recognition model;The Chinese speech data to be identified is input into the target Chinese speech recognition model, and Chinese speech recognition result is obtained.The present application adds the loss calculation of multilevel multi-granularity, so that CTC coding network can extract more rich and varied speech feature information, while not affecting model inference speed and model complexity, improve the accuracy of Chinese speech recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, and particularly relates to a Chinese speech recognition method and system, a storage medium and an electronic device. BACKGROUND

[0002] In current Chinese speech recognition models, there are generally problems of low recognition accuracy or high model complexity. If a complex encoding network is constructed only to improve the recognition accuracy of the model, the calculation difficulty will undoubtedly be greatly increased, thereby increasing the model inference speed.

[0003] How to improve the model accuracy while not reducing the model recognition efficiency is a technical problem to be solved at present. SUMMARY

[0004] To solve the above technical problems, the present application provides a Chinese speech recognition method and system, a storage medium and an electronic device.

[0005] The technical scheme of the Chinese speech recognition method of the present application is as follows:

[0006] Based on a plurality of Chinese speech training samples, an original CTC encoding network to which a fine-grained loss module and two intermediate layer loss modules are added is trained to obtain a first Chinese speech recognition model, and the fine-grained loss module and the two intermediate layer loss modules in the first Chinese speech recognition model are deleted to obtain a target Chinese speech recognition model; wherein the fine-grained loss module is used to receive and obtain a fine-grained loss value according to a first Chinese speech feature output by a previous original encoding layer; any intermediate layer loss module is used to receive and process a second Chinese speech feature output by a previous original encoding layer to obtain a third Chinese speech feature, and the third Chinese speech feature is spliced with the second Chinese speech feature to obtain a spliced Chinese speech feature which is input into a next original encoding layer;

[0007] The Chinese speech recognition result is obtained by inputting the to-be-recognized Chinese speech data into the target Chinese speech recognition model.

[0008] The Chinese speech recognition method of the present application has the following beneficial effects:

[0009] The method of the present application adds multiple levels and multiple granularities of loss calculation, so that the CTC encoding network can extract more diverse speech feature information, and improves the accuracy of Chinese speech recognition without affecting the model inference speed and model complexity.

[0010] On the basis of the above scheme, the Chinese speech recognition method of the present application can be further improved as follows.

[0011] Further, the fine-grained loss module comprises: a first normalization layer, a first linear layer and a first softmax function arranged in sequence, and a fine-grained loss function.

[0012] The fine-grained loss module is specifically configured to: receive first Chinese speech features output by a previous encoding layer corresponding to the fine-grained loss module, and sequentially process the first Chinese speech features through the first normalization layer, the first linear layer and the first softmax function to obtain processed first Chinese speech features, and input the processed first Chinese speech features into the fine-grained loss function for calculation to obtain the fine-grained loss value.

[0013] Further, the two intermediate layer loss modules comprise: a pinyin level intermediate layer loss module comprising a second normalization layer, a second linear layer, a second softmax function, a first newly added linear layer and a pinyin level loss function arranged in sequence, and a character level intermediate layer loss module comprising a third normalization layer, a third linear layer, a third softmax function, a second newly added linear layer and a character level loss function arranged in sequence.

[0014] The pinyin level intermediate layer loss module is configured to: receive second Chinese speech features output by a previous original encoding layer corresponding to the pinyin level intermediate layer loss module, and sequentially process the second Chinese speech features of the pinyin level intermediate layer loss module through the second normalization layer, the second linear layer and the second softmax function to obtain third Chinese speech features of the pinyin level intermediate layer loss module, and splice the third Chinese speech features of the pinyin level intermediate layer loss module with the second Chinese speech features of the pinyin level intermediate layer loss module through the first newly added linear layer to obtain spliced Chinese speech features of the pinyin level intermediate layer loss module and output the spliced Chinese speech features to a next original encoding layer corresponding to the pinyin level intermediate layer loss module.

[0015] The character level intermediate layer loss module is configured to: receive second Chinese speech features output by a previous original encoding layer corresponding to the character level intermediate layer loss module, and sequentially process the second Chinese speech features of the character level intermediate layer loss module through the third normalization layer, the third linear layer and the third softmax function to obtain third Chinese speech features of the character level intermediate layer loss module, and splice the third Chinese speech features of the character level intermediate layer loss module with the second Chinese speech features of the character level intermediate layer loss module through the second newly added linear layer to obtain spliced Chinese speech features of the character level intermediate layer loss module and output the spliced Chinese speech features to a next original encoding layer corresponding to the character level intermediate layer loss module.

[0016] The parameters of the first normalization layer and the second normalization layer are the same, and the parameters of the first linear layer and the second linear layer are the same; the parameters of the third normalization layer are the same as those of the original normalization layer of the original CTC encoding network, and the parameters of the third linear layer are the same as those of the original linear layer of the original CTC encoding network.

[0017] Further, the pinyin-level intermediate layer loss module is further configured to input the third Chinese speech feature of the pinyin-level intermediate layer loss module into the pinyin-level loss function for calculation to obtain a first pinyin-level loss value.

[0018] The character-level intermediate layer loss module is further configured to input the third Chinese speech feature of the character-level intermediate layer loss module into the character-level loss function for calculation to obtain a first character-level loss value.

[0019] Further, the original CTC encoding network added with one fine-grained loss module and two intermediate layer loss modules is trained based on a plurality of Chinese speech training samples to obtain a first Chinese speech recognition model, including:

[0020] Each Chinese speech training sample is input into the original CTC encoding network added with one fine-grained loss module and two intermediate layer loss modules for training, and a target loss value of each Chinese speech training sample is obtained according to the fine-grained loss value, the first pinyin-level loss value, the first character-level loss value and the original CTC network loss value of each Chinese speech training sample;

[0021] The parameters of the original CTC encoding network added with one fine-grained loss module and two intermediate layer loss modules are optimized according to all target loss values to obtain an optimized CTC encoding network, and the optimized CTC encoding network is taken as the original CTC encoding network, and the step of training the original CTC encoding network added with one fine-grained loss module and two intermediate layer loss modules based on a plurality of Chinese speech training samples is executed until the optimized CTC encoding network meets a preset condition, and the first Chinese speech recognition model is obtained.

[0022] Further, the step of obtaining the target loss value of each Chinese speech training sample according to the fine-grained loss value, the first pinyin-level loss value, the first character-level loss value and the original CTC network loss value of each Chinese speech training sample includes:

[0023] The target pinyin-level loss value of any Chinese speech training sample is obtained based on a first preset formula, the fine-grained loss value and the first pinyin-level loss value of the Chinese speech training sample; wherein the first preset formula is: A first pinyin level loss value of any Chinese speech training sample, A fine-grained loss value of any Chinese speech training sample, Lpinyin-CTC is a target pinyin level loss value of any Chinese speech training sample, and a is a weight coefficient of a pinyin level auxiliary loss, a∈(0, 1);

[0024] Based on a second preset formula, a first character level loss value of any Chinese speech training sample and an original CTC network loss value, a target character level loss value of any Chinese speech training sample is obtained; wherein the second preset formula is: A first character level loss value of any Chinese speech training sample, An original CTC network loss value of any Chinese speech training sample, L c h aracter-CTC A target character level loss value of any Chinese speech training sample, and b is a weight coefficient of a character level auxiliary loss, b∈(0, 1);

[0025] Based on a third preset formula, a target pinyin level loss value of any Chinese speech training sample and a target character level loss value, a target loss value of any Chinese speech training sample is obtained, until a target loss value of each Chinese training sample is obtained; wherein the third preset formula is: L=(1-λ)L c h aracter-CTC +λL pinyin-CTC , and L is a target loss value of any Chinese speech training sample, and λ∈(0, 1).

[0026] The technical scheme of a Chinese speech recognition system of the present application is as follows:

[0027] It comprises a training module and a recognition module.

[0028] The training module is used to train an original CTC encoding network added with a fine-grained loss module and two intermediate layer loss modules based on a plurality of Chinese speech training samples, obtain a first Chinese speech recognition model, and delete the fine-grained loss module and the two intermediate layer loss modules in the first Chinese speech recognition model to obtain a target Chinese speech recognition model; wherein the fine-grained loss module is used to receive and obtain a fine-grained loss value according to a first Chinese speech feature output by a previous original encoding layer; any intermediate layer loss module is used to receive and process a second Chinese speech feature output by a previous original encoding layer to obtain a third Chinese speech feature, splice the third Chinese speech feature with the second Chinese speech feature, and input the spliced Chinese speech feature into a next original encoding layer;

[0029] The identification module is configured to input the to-be-identified Chinese speech data into the target Chinese speech recognition model to obtain a Chinese speech recognition result.

[0030] The Chinese speech recognition system has the following advantages:

[0031] The system adds a multi-level and multi-granularity loss calculation, so that the CTC encoding network can extract more diverse speech feature information, improves the accuracy of Chinese speech recognition without affecting the inference speed of the model and the complexity of the model.

[0032] On the basis of the above-mentioned scheme, the Chinese speech recognition system of the present application can also be improved as follows.

[0033] Further, the fine-grained loss module comprises a first normalization layer, a first linear layer and a first softmax function and a fine-grained loss function arranged in sequence.

[0034] The fine-grained loss module is specifically configured to receive the first Chinese speech feature output by the previous encoding layer corresponding to the fine-grained loss module, and sequentially process the first Chinese speech feature through the first normalization layer, the first linear layer and the first softmax function, obtain the processed first Chinese speech feature and input it into the fine-grained loss function for calculation to obtain the fine-grained loss value.

[0035] The technical scheme of the storage medium of the present application is as follows:

[0036] The storage medium stores instructions, and when a computer reads the instructions, the computer executes the steps of the Chinese speech recognition method of the present application.

[0037] The technical scheme of the electronic device of the present application is as follows:

[0038] The electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to make the computer execute the steps of the Chinese speech recognition method of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 The flowchart of the Chinese speech recognition method of the embodiment of the present application is shown in the figure.

[0040] Figure 2 The structure diagram of the original CTC encoding network in the Chinese speech recognition method of the embodiment of the present application is shown in the figure.

[0041] Figure 3This is a schematic diagram of the original CTC coding network with one fine-grained loss module and two intermediate layer loss modules added in a Chinese speech recognition method according to an embodiment of the present invention.

[0042] Figure 4 This is a schematic diagram of the structure of a Chinese speech recognition system according to an embodiment of the present invention. Detailed Implementation

[0043] like Figure 1 As shown, an embodiment of the present invention provides a Chinese speech recognition method, which includes the following steps:

[0044] S1. Based on multiple Chinese speech training samples, train the original CTC encoding network with one fine-grained loss module and two intermediate layer loss modules to obtain a first Chinese speech recognition model, and then delete the fine-grained loss module and the two intermediate layer loss modules from the first Chinese speech recognition model to obtain the target Chinese speech recognition model.

[0045] The fine-grained loss module is used to: receive and obtain a fine-grained loss value based on the first Chinese speech feature output by the previous coding layer; any intermediate layer loss module is used to: receive and process the second Chinese speech feature output by the previous original coding layer to obtain a third Chinese speech feature, and concatenate the third Chinese speech feature with the second Chinese speech feature to obtain and input the concatenated Chinese speech feature into the next original coding layer.

[0046] The Chinese speech training samples are Chinese speech data collected in any way, and the number of Chinese speech training samples is unlimited.

[0047] Among them, such as Figure 2 As shown, the original CTC coding network includes: 12 layers of original coding layers, original normalization layers, original linear layers, original softmax function, and original loss function arranged sequentially.

[0048] The fine-grained loss module and each intermediate layer loss model are respectively set between any two adjacent original coding layers. The two intermediate layer loss models include: a pinyin-level intermediate layer loss module and a character-level intermediate layer loss module. For example, Figure 3As shown, in the embodiment, the fine-grained loss module is arranged between the 6th original encoding layer and the 7th original encoding layer, the pinyin-level intermediate layer loss module is arranged between the 3rd original encoding layer and the 4th original encoding layer, and the character-level intermediate layer loss module is arranged between the 9th original encoding layer and the 10th original encoding layer. The previous encoding layer corresponding to the fine-grained loss module is the 6th original encoding layer, and the next encoding layer corresponding to the fine-grained loss module is the 7th original encoding layer. The previous encoding layer corresponding to the pinyin-level intermediate layer loss module is the 3rd original encoding layer, and the next encoding layer corresponding to the pinyin-level intermediate layer loss module is the 4th original encoding layer. The previous encoding layer corresponding to the character-level intermediate layer loss module is the 9th original encoding layer, and the next encoding layer corresponding to the character-level intermediate layer loss module is the 10th original encoding layer.

[0049] The first Chinese speech recognition model is a model obtained by training the improved original CTC encoding network (the original CTC encoding network to which a fine-grained loss module and two intermediate layer loss modules are added) using a plurality of Chinese speech training samples. The target Chinese speech recognition model is a model obtained by deleting the fine-grained loss module and the two intermediate layer loss modules in the first Chinese speech recognition model.

[0050] The first Chinese speech feature is a Chinese speech feature output by the previous original encoding layer corresponding to the fine-grained loss module. The second Chinese speech feature is a Chinese speech feature output by the previous original encoding layer corresponding to any intermediate layer loss module (the pinyin-level intermediate layer loss module or the character-level intermediate layer loss module). The third Chinese speech feature is a Chinese speech feature obtained after a series of processing by any intermediate layer loss module. The spliced Chinese speech feature is a Chinese speech feature obtained by splicing a plurality of speech features.

[0051] It should be noted that in the embodiment, the fbank feature needs to be extracted from the Chinese speech training sample, and then the fbank feature of each Chinese speech training sample is input into the model for training.

[0052] It should be noted that any original encoding layer in the original CTC encoding network includes a feedforward module (FeedForward Module), a multi-head attention module (Multi-Head Self Attention Module), and a convolution module (Convolution Module). Each original encoding layer is configured to: sequentially pass the input speech feature through the feedforward module, the multi-head attention module, and the convolution module for processing, and output the processed speech feature (the first Chinese speech feature in the embodiment). The specific process of processing the speech feature by the feedforward module, the multi-head attention module, and the convolution module is a prior art, which will not be described in detail here.

[0053] S2, input the to-be-recognized Chinese voice data into the target Chinese voice recognition model to obtain a Chinese voice recognition result.

[0054] The to-be-recognized Chinese voice data is any collected voice data. Similarly, the fbank features of the to-be-recognized Chinese voice data need to be extracted, and the fbank features of the to-be-recognized Chinese voice data are input into the target voice recognition model to obtain the Chinese voice recognition result of the to-be-recognized Chinese voice data.

[0055] The Chinese voice recognition result is the text content corresponding to the to-be-recognized Chinese voice data.

[0056] Preferably, the fine-grained loss module comprises a first normalization layer, a first linear layer and a first softmax function and a fine-grained loss function arranged in sequence.

[0057] The fine-grained loss module is specifically configured to receive the first Chinese voice features output by the previous layer of the fine-grained loss module, and sequentially process the first Chinese voice features through the first normalization layer, the first linear layer and the first softmax function to obtain the processed first Chinese voice features and input the processed first Chinese voice features into the fine-grained loss function for calculation to obtain the fine-grained loss value.

[0058] It should be noted that the process of normalizing the voice features through the normalization layer, the process of linearly processing the voice features through the linear layer, and the process of processing the voice features through the softmax function are all prior art, and will not be described in detail here.

[0059] Preferably, the two intermediate layer loss modules comprise a pinyin level intermediate layer loss module comprising a second normalization layer, a second linear layer, a second softmax function, a first additional linear layer and a pinyin level loss function arranged in sequence, and a character level intermediate layer loss module comprising a third normalization layer, a third linear layer, a third sotfmax function, a second additional linear layer and a character level loss function arranged in sequence.

[0060] The pinyin level intermediate layer loss module comprises a second normalization layer, a second linear layer, a second softmax function, a first additional linear layer and a pinyin level loss function arranged in sequence; and the character level intermediate layer loss module comprises a third normalization layer, a third linear layer, a third sotfmax function, a second additional linear layer and a character level loss function arranged in sequence.

[0061] It should be noted that the parameters of the first normalization layer and the second normalization layer are the same, and the parameters of the first linear layer and the second linear layer are the same; the parameters of the third normalization layer and the original normalization layer of the original CTC encoding network are the same, and the parameters of the third linear layer and the original linear layer of the original CTC encoding network are the same.

[0062] The pinyin-level intermediate layer loss module is configured to: receive second Chinese speech features output by a previous layer of an original encoding layer corresponding to the pinyin-level intermediate layer loss module, and sequentially process the second Chinese speech features of the pinyin-level intermediate layer loss module through the second normalization layer, the second linear layer, and the second softmax function to obtain third Chinese speech features of the pinyin-level intermediate layer loss module, and concatenate the third Chinese speech features of the pinyin-level intermediate layer loss module with the second Chinese speech features of the pinyin-level intermediate layer loss module through the first newly added linear layer to obtain and output spliced Chinese speech features of the pinyin-level intermediate layer loss module to a next layer of the original encoding layer corresponding to the pinyin-level intermediate layer loss module.

[0063] Specifically, based on the fourth preset formula, the second Chinese speech features of the pinyin-level intermediate layer loss module are sequentially processed through the second normalization layer, the second linear layer, and the second softmax function to obtain the third Chinese speech features of the pinyin-level intermediate layer loss module.

[0064] The fourth preset formula is:

[0065] The third Chinese speech features of the pinyin-level intermediate layer loss module are: The second Chinese speech features of the pinyin-level intermediate layer loss module are: The second Chinese speech features of the pinyin-level intermediate layer loss module are normalized by the second normalization layer. The second Chinese speech features of the pinyin-level intermediate layer loss module are linearly transformed by the second linear layer. The feature dimension of is mapped from V to D. The feature dimension of is mapped from V to D.

[0066] Based on the fifth preset formula and the sixth preset formula, the third Chinese speech features of the pinyin-level intermediate layer loss module are concatenated with the second Chinese speech features of the pinyin-level intermediate layer loss module through the first newly added linear layer to obtain and output spliced Chinese speech features of the pinyin-level intermediate layer loss module to a next layer of the original encoding layer corresponding to the pinyin-level intermediate layer loss module.

[0067] wherein the fifth preset formula is: is mapped from V to D through the first newly added linear layer, is the third Chinese speech feature of the pinyin-level intermediate layer loss module after feature dimension mapping.

[0068] wherein the sixth preset formula is: is the input feature of the next layer original encoding layer corresponding to the pinyin-level intermediate layer loss module.

[0069] It should be noted that the feature dimension of is mapped from V to D through the first newly added linear layer, so as to keep the feature dimensions of the second Chinese speech feature and the third Chinese speech feature of the pinyin-level intermediate layer loss module after feature dimension mapping consistent.

[0070] In this embodiment, the pinyin-level intermediate layer loss module receives the second Chinese speech feature output by the third layer original encoding layer, and sequentially processes the second Chinese speech feature of the pinyin-level intermediate layer loss module through the second normalization layer, the second linear layer and the second softmax function to obtain the third Chinese speech feature of the pinyin-level intermediate layer loss module, and splices the third Chinese speech feature of the pinyin-level intermediate layer loss module and the second Chinese speech feature of the pinyin-level intermediate layer loss module through the first newly added linear layer to obtain the spliced Chinese speech feature of the pinyin-level intermediate layer loss module, and outputs the spliced Chinese speech feature to the fourth layer original encoding layer.

[0071] The character-level intermediate layer loss module is configured to: receive the second Chinese speech feature output by the original encoding layer corresponding to the character-level intermediate layer loss module, and sequentially process the second Chinese speech feature of the character-level intermediate layer loss module through the third normalization layer, the third linear layer and the third softmax function to obtain the third Chinese speech feature of the character-level intermediate layer loss module, and splice the third Chinese speech feature of the character-level intermediate layer loss module and the second Chinese speech feature of the character-level intermediate layer loss module through the second newly added linear layer to obtain the spliced Chinese speech feature of the character-level intermediate layer loss module and output the spliced Chinese speech feature to the original encoding layer corresponding to the character-level intermediate layer loss module.

[0072] ​Specifically, the character-level intermediate layer loss module receives the second Chinese speech features of the 9th layer original encoding layer output, and sequentially processes the second Chinese speech features of the character-level intermediate layer loss module through a third normalization layer, a third linear layer, and a third softmax function to obtain third Chinese speech features of the character-level intermediate layer loss module, and splices the third Chinese speech features of the character-level intermediate layer loss module with the second Chinese speech features of the character-level intermediate layer loss module through a second newly added linear layer to obtain spliced Chinese speech features of the character-level intermediate layer loss module, and outputs the spliced Chinese speech features to the 10th layer original encoding layer.

[0073] It should be noted that the calculation formula of the specific process in the character-level intermediate layer loss module is the same as the calculation formula described in the pinyin-level intermediate layer loss module, which only needs to be adjusted accordingly, and will not be repeated here.

[0074] Preferably, the pinyin-level intermediate layer loss module is further configured to input the third Chinese speech features of the pinyin-level intermediate layer loss module into the pinyin-level loss function for calculation to obtain a first pinyin-level loss value.

[0075] The first pinyin-level loss value is a loss value calculated according to the Chinese speech features output by the 3rd layer original encoding layer.

[0076] The character-level intermediate layer loss module is further configured to input the third Chinese speech features of the character-level intermediate layer loss module into the character-level loss function for calculation to obtain a first character-level loss value.

[0077] The first character-level loss value is a loss value calculated according to the Chinese speech features output by the 9th layer original encoding layer.

[0078] Preferably, the first Chinese speech recognition model is obtained by training the original CTC encoding network added with one fine-grained loss module and two intermediate layer loss modules based on a plurality of Chinese speech training samples, including:

[0079] Each Chinese speech training sample is input into the original CTC encoding network added with one fine-grained loss module and two intermediate layer loss modules for training, and a target loss value of each Chinese speech training sample is obtained according to the fine-grained loss value, the first pinyin-level loss value, the first character-level loss value, and the original CTC network loss value of each Chinese speech training sample.

[0080] The original CTC network loss value is a loss value calculated according to an original loss function when the Chinese speech training sample is input into the original CTC encoding network added with one fine-grained loss module and two intermediate layer loss modules for training.

[0081] The target loss value is a loss value obtained by weighted summation of a plurality of loss values corresponding to the Chinese speech training samples.

[0082] The parameters of the original CTC encoding network added with one fine-grained loss module and two intermediate layer loss modules are optimized according to all target loss values, and the optimized CTC encoding network is obtained and taken as the original CTC encoding network. The step of training the original CTC encoding network added with one fine-grained loss module and two intermediate layer loss modules based on a plurality of Chinese speech training samples is executed until the optimized CTC encoding network meets a preset condition, and the first Chinese speech recognition model is obtained.

[0083] The preset condition is any one of the following modes: ① reaching an expected index (word accuracy, which is an existing general index) requirement; ② model loss converging to a specified threshold and no longer decreasing; and ③ reaching a specified maximum number of training times.

[0084] Specifically, the parameters of the original CTC encoding network added with one fine-grained loss module and two intermediate layer loss modules are reversely optimized according to all target loss values, and the optimized CTC encoding network is obtained. It is judged whether the optimized CTC encoding network meets a preset condition. If yes, the optimized CTC encoding network is determined as the first Chinese speech recognition model; if no, the optimized CTC encoding network is taken as the original CTC encoding network, and the step S1 is executed until the optimized CTC encoding network meets the preset condition, and the first Chinese speech recognition model is obtained.

[0085] Preferably, the step of obtaining the target loss value of each Chinese speech training sample according to the fine-grained loss value, the first pinyin level loss value, the first character level loss value and the original CTC network loss value of each Chinese speech training sample comprises:

[0086] The target pinyin level loss value of any Chinese speech training sample is obtained based on a first preset formula, the fine-grained loss value and the first pinyin level loss value of the any Chinese speech training sample.

[0087] The first preset formula is: The first pinyin level loss value of the any Chinese speech training sample, The fine-grained loss value of the any Chinese speech training sample, Lpinyin-CTC is the target pinyin level loss value of the any Chinese speech training sample, and a is a weight coefficient of a pinyin level auxiliary loss, a ∈ (0, 1).

[0088] The target character-level loss value of the any Chinese speech training sample is obtained based on a second preset formula, the first character-level loss value and the original CTC network loss value of the any Chinese speech training sample.

[0089] The second preset formula is: The first character-level loss value of the any Chinese speech training sample, The original CTC network loss value of the any Chinese speech training sample is L c h aracter-CTC The target character-level loss value of the any Chinese speech training sample, and β is a weight coefficient of the character-level auxiliary loss, and β ∈ (0, 1).

[0090] The target loss value of the any Chinese speech training sample is obtained based on a third preset formula, the target pinyin-level loss value and the target character-level loss value of the any Chinese speech training sample, until the target loss value of each Chinese training sample is obtained.

[0091] The third preset formula is: L = (1-λ)L c h aracter-CTC +λL pinyin-CTC The target loss value of the any Chinese speech training sample is L, and λ ∈ (0, 1).

[0092] It should be noted that the fine-grained loss value and the first pinyin-level loss value are obtained according to the difference between the predicted value obtained by the corresponding module and the pinyin-level real label (true value) in Figure 3 For example, the pinyin-level real label is “jin tianxing qi wu”. The first character-level loss value and the original CTC network loss value are obtained according to the difference between the predicted value obtained by the corresponding module and the character-level real label (true value) in Figure 3 For example, the character-level real label is “today Friday”.

[0093] The technical scheme of the embodiment adds multi-level and multi-granularity loss calculation, so that the CTC encoding network can extract more rich and diverse speech feature information, improves the accuracy of Chinese speech recognition without affecting the inference speed of the model and the complexity of the model.

[0094] As shown in Figure 4 A speech recognition system 200 according to an embodiment of the present application includes a training module 210 and a recognition module 220.

[0095] The training module 210 is used to: train an original CTC encoding network with one fine-grained loss module and two intermediate layer loss modules added based on multiple Chinese speech training samples to obtain a first Chinese speech recognition model, and then delete the fine-grained loss module and the two intermediate layer loss modules from the first Chinese speech recognition model to obtain a target Chinese speech recognition model; wherein, the fine-grained loss module is used to: receive and obtain a fine-grained loss value based on the first Chinese speech feature output by the previous original encoding layer; any intermediate layer loss module is used to: receive and process the second Chinese speech feature output by the previous original encoding layer to obtain a third Chinese speech feature, and concatenate the third Chinese speech feature with the second Chinese speech feature to obtain and input the concatenated Chinese speech feature into the next original encoding layer;

[0096] The recognition module 220 is used to: input the Chinese speech data to be recognized into the target Chinese speech recognition model to obtain the Chinese speech recognition result.

[0097] Preferably, the fine-grained loss module includes: a first normalization layer, a first linear layer, a first softmax function, and a fine-grained loss function arranged sequentially;

[0098] The fine-grained loss module is specifically used to: receive the first Chinese speech feature output by the previous coding layer corresponding to the fine-grained loss module, and process it sequentially through the first normalization layer, the first linear layer and the first softmax function to obtain the first Chinese speech feature. The processed first Chinese speech feature is then input into the fine-grained loss function for calculation to obtain the fine-grained loss value.

[0099] The technical solution in this embodiment adds multi-level, multi-granularity loss calculations, enabling the CTC coding network to extract richer and more diverse speech feature information. This improves the accuracy of Chinese speech recognition without affecting the model's inference speed and complexity.

[0100] The parameters and steps for implementing the corresponding functions of each module in the Chinese speech recognition system 200 described above in this embodiment can be referred to the parameters and steps in the embodiment of the Chinese speech recognition method above, and will not be repeated here.

[0101] An embodiment of the present invention provides a storage medium, comprising: the storage medium storing instructions, which, when a computer reads the instructions, cause the computer to execute steps such as those of a Chinese speech recognition system method. For details, please refer to the parameters and steps in the embodiment of the Chinese speech recognition system method above, which will not be repeated here.

[0102] Computer storage media, such as USB flash drives and external hard drives.

[0103] An electronic device provided by an embodiment of the present application includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the computer performs steps of a method for recognizing Chinese speech. Details can be found in the embodiments of the method for recognizing Chinese speech, which will not be repeated here.

[0104] Those skilled in the art know that the present application can be implemented as a method, a system, a storage medium, and an electronic device.

[0105] Therefore, the present application can be embodied in the form of entirely hardware, entirely software (including firmware, resident software, micro-code, etc.), or a combination of hardware and software, which can be referred to as a "circuit", "module" or "system" herein. Furthermore, in some embodiments, the present application can also be embodied in the form of a computer program product including a computer readable medium having stored thereon a computer readable program code. Any combination of one or more computer readable medium can be used. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer readable storage medium can be any tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary, and are not to be interpreted as limiting the present application, and those skilled in the art can make various changes, modifications, replacements, and variations to the above embodiments within the scope of the present application.

Claims

1. A method for Chinese speech recognition, the method comprising: The method comprises the following steps: training an original CTC encoding network added with a fine-grained loss module and two intermediate layer loss modules based on a plurality of Chinese speech training samples to obtain a first Chinese speech recognition model, and deleting the fine-grained loss module and the two intermediate layer loss modules in the first Chinese speech recognition model to obtain a target Chinese speech recognition model; wherein the fine-grained loss module is used for receiving and obtaining a fine-grained loss value according to first Chinese speech features output by a previous original encoding layer; any intermediate layer loss module is used for receiving and processing second Chinese speech features output by a previous original encoding layer to obtain third Chinese speech features, splicing the third Chinese speech features with the second Chinese speech features, and inputting the spliced Chinese speech features into a next original encoding layer; inputting to-be-recognized Chinese speech data into the target Chinese speech recognition model to obtain a Chinese speech recognition result; The original CTC encoding network comprises 12 layers of original encoding layers, an original normalization layer, an original linear layer, an original softmax function, and an original loss function arranged in sequence; the fine-grained loss module and each intermediate layer loss model are arranged between any two adjacent original encoding layers, respectively; the two intermediate layer loss models comprise a pinyin-level intermediate layer loss module and a character-level intermediate layer loss module; the fine-grained loss module is arranged between the 6th original encoding layer and the 7th original encoding layer, the pinyin-level intermediate layer loss module is arranged between the 3rd original encoding layer and the 4th original encoding layer, and the character-level intermediate layer loss module is arranged between the 9th original encoding layer and the 10th original encoding layer.

2. The Chinese speech recognition method of claim 1, wherein, The fine-grained loss module comprises a first normalization layer, a first linear layer, a first softmax function, and a fine-grained loss function arranged in sequence; The fine-grained loss module is specifically used for receiving first Chinese speech features output by a previous layer of encoding layers corresponding to the fine-grained loss module, processing the first Chinese speech features in sequence through the first normalization layer, the first linear layer, and the first softmax function, inputting the processed first Chinese speech features into the fine-grained loss function for calculation, and obtaining the fine-grained loss value.

3. The Chinese speech recognition method of claim 2, wherein, The two intermediate layer loss modules comprise a pinyin-level intermediate layer loss module comprising a second normalization layer, a second linear layer, a second softmax function, a first newly added linear layer, and a pinyin-level loss function arranged in sequence, and a character-level intermediate layer loss module comprising a third normalization layer, a third linear layer, a third softmax function, a second newly added linear layer, and a character-level loss function arranged in sequence. The pinyin-level intermediate layer loss module is configured to receive the second Chinese speech feature output by the previous original encoding layer corresponding to the pinyin-level intermediate layer loss module, and sequentially process the second Chinese speech feature of the pinyin-level intermediate layer loss module through the second normalization layer, the second linear layer and the second softmax function to obtain a third Chinese speech feature of the pinyin-level intermediate layer loss module, and splice the third Chinese speech feature of the pinyin-level intermediate layer loss module with the second Chinese speech feature of the pinyin-level intermediate layer loss module through the first newly-added linear layer to obtain a spliced Chinese speech feature of the pinyin-level intermediate layer loss module and output the spliced Chinese speech feature to the next original encoding layer corresponding to the pinyin-level intermediate layer loss module. The character-level intermediate layer loss module is configured to receive the second Chinese speech feature output by the previous original encoding layer corresponding to the character-level intermediate layer loss module, and sequentially process the second Chinese speech feature of the character-level intermediate layer loss module through the third normalization layer, the third linear layer and the third softmax function to obtain a third Chinese speech feature of the character-level intermediate layer loss module, and splice the third Chinese speech feature of the character-level intermediate layer loss module with the second Chinese speech feature of the character-level intermediate layer loss module through the second newly-added linear layer to obtain a spliced Chinese speech feature of the character-level intermediate layer loss module and output the spliced Chinese speech feature to the next original encoding layer corresponding to the character-level intermediate layer loss module. The parameters of the first normalization layer and the second normalization layer are the same, and the parameters of the first linear layer and the second linear layer are the same; the parameters of the third normalization layer are the same as those of the original normalization layer of the original CTC encoding network, and the parameters of the third linear layer are the same as those of the original linear layer of the original CTC encoding network.

4. The Chinese speech recognition method of claim 3, wherein, The pinyin-level intermediate layer loss module is further configured to input the third Chinese speech feature of the pinyin-level intermediate layer loss module into the pinyin-level loss function for calculation to obtain a first pinyin-level loss value. The character-level intermediate layer loss module is further configured to input the third Chinese speech feature of the character-level intermediate layer loss module into the character-level loss function for calculation to obtain a first character-level loss value.

5. The Chinese speech recognition method of claim 4, wherein, The training of the original CTC encoding network added with one fine-grained loss module and two intermediate layer loss modules based on a plurality of Chinese speech training samples to obtain a first Chinese speech recognition model includes: Each Chinese speech training sample is input into the original CTC encoding network added with one fine-grained loss module and two intermediate layer loss modules for training to obtain a target loss value of each Chinese speech training sample according to the fine-grained loss value, the first pinyin-level loss value, the first character-level loss value and the original CTC network loss value of each Chinese speech training sample. Each Chinese speech training sample is input into the original CTC encoding network added with one fine-grained loss module and two intermediate layer loss modules for training to obtain a target loss value of each Chinese speech training sample according to the fine-grained loss value, the first pinyin-level loss value, the first character-level loss value and the original CTC network loss value of each Chinese speech training sample. According to all the target loss values, the parameters of the original CTC encoding network added with one fine-grained loss module and two intermediate layer loss modules are optimized, and the optimized CTC encoding network is obtained and taken as the original CTC encoding network, and the step of training the original CTC encoding network added with one fine-grained loss module and two intermediate layer loss modules based on multiple Chinese speech training samples is executed until the optimized CTC encoding network meets a preset condition, and the first Chinese speech recognition model is obtained.

6. The Chinese speech recognition method of claim 5, wherein, The step of obtaining the target loss value of each Chinese speech training sample according to the fine-grained loss value, the first pinyin level loss value, the first character level loss value and the original CTC network loss value of each Chinese speech training sample comprises: obtaining a target pinyin level loss value of any Chinese speech training sample based on a first preset formula, a fine-grained loss value of the any Chinese speech training sample and a first pinyin level loss value of the any Chinese speech training sample; wherein the first preset formula is: , is the first pinyin level loss value of the any Chinese speech training sample, is the fine-grained loss value of the any Chinese speech training sample, is the target pinyin level loss value of the any Chinese speech training sample, is a weight coefficient of the pinyin level auxiliary loss, ; obtaining a target character-level loss value of the any Chinese speech training sample based on a second preset formula, the first character-level loss value of the any Chinese speech training sample and the original CTC network loss value; , is the first character-level loss value of the any Chinese speech training sample, is the original CTC network loss value of the any Chinese speech training sample, is the target character-level loss value of the any Chinese speech training sample, is a weight coefficient of the character-level auxiliary loss, ; Based on the third preset formula, the target pinyin level loss value and the target character level loss value of the any Chinese speech training sample, a target loss value of the any Chinese speech training sample is obtained, until a target loss value of each Chinese training sample is obtained; wherein the third preset formula is: , is the target loss value of the any Chinese speech training sample, .

7. A Chinese speech recognition system, characterized in that, It comprises: The training module and the recognition module; The training module is used for training the original CTC encoding network added with one fine-grained loss module and two intermediate layer loss modules based on multiple Chinese speech training samples, obtaining a first Chinese speech recognition model, and deleting the fine-grained loss module and the two intermediate layer loss modules in the first Chinese speech recognition model to obtain a target Chinese speech recognition model; wherein the fine-grained loss module is used for receiving and obtaining a fine-grained loss value according to the first Chinese speech feature output by the previous original encoding layer; any intermediate layer loss module is used for receiving and processing the second Chinese speech feature output by the previous original encoding layer to obtain a third Chinese speech feature, and splicing the third Chinese speech feature and the second Chinese speech feature to obtain and input the spliced Chinese speech feature into the next original encoding layer; The recognition module is used for inputting the Chinese speech data to be recognized into the target Chinese speech recognition model to obtain a Chinese speech recognition result; The original CTC encoding network comprises 12 original encoding layers, an original normalization layer, an original linear layer, an original softmax function and an original loss function arranged in sequence; the fine-grained loss module and each intermediate layer loss model are arranged between any two adjacent original encoding layers respectively; the two intermediate layer loss models comprise a pinyin level intermediate layer loss module and a character level intermediate layer loss module; the fine-grained loss module is arranged between the 6th original encoding layer and the 7th original encoding layer, the pinyin level intermediate layer loss module is arranged between the 3rd original encoding layer and the 4th original encoding layer, and the character level intermediate layer loss module is arranged between the 9th original encoding layer and the 10th original encoding layer.

8. The Chinese speech recognition system of claim 7, wherein, The fine-grained loss module comprises a first normalization layer, a first linear layer, a first softmax function and a fine-grained loss function arranged in sequence; The fine-grained loss module is specifically configured to receive a first Chinese speech feature output by a previous coding layer corresponding to the fine-grained loss module, and sequentially process the first Chinese speech feature through the first normalization layer, the first linear layer and the first softmax function to obtain a processed first Chinese speech feature, and input the processed first Chinese speech feature into the fine-grained loss function to calculate the fine-grained loss value.

9. A storage medium, characterized by The storage medium has instructions stored therein, and when a computer reads the instructions, the computer executes a Chinese speech recognition method according to any one of claims 1 to 6.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program, and the computer executes a Chinese speech recognition method according to any one of claims 1 to 6.

Images