Model training method and device applied to speech recognition, equipment and storage medium

Abstract

The embodiment of the application belongs to the technical field of speech recognition in artificial intelligence, and relates to a model training method and device applied to speech recognition, a computer device and a storage medium. The embodiment of the application combines an IMV-based loss function in a joint loss function, can drive a model to learn an alignment relationship between speech and characters explicitly and quickly, thereby accelerating the training speed. The alignment constraint of the IMV-based loss function does not need to change the model structure, is easy to implement, has small code engineering quantity, and is suitable for any explicit or implicit use of an attention mechanism and a monotonic alignment model, such as speech recognition, speech synthesis and the like, and has universality and migratability. Meanwhile, because of the more accurate monotonic alignment relationship, the model can learn better representation capability under limited data, thereby effectively improving the recognition accuracy of the model.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology in artificial intelligence, and in particular to a model training method, apparatus, computer equipment and storage medium for speech recognition. Background Art

[0002] Automatic speech recognition (ASR) is a technology that transcribes speech signals into corresponding text. It is widely used in products and scenarios such as voice input methods, intelligent customer service, smart speakers, and in-car voice assistants. With the rapid development of artificial intelligence and the continuous iteration of deep neural network technology, speech recognition has evolved from complex hybrid models to end-to-end models. End-to-end models are not only easier to develop and deploy, but also offer higher accuracy. Among them, end-to-end models based on attention mechanisms represent the cutting-edge state of the art in speech recognition. There are many types of attention mechanisms, the most commonly used in speech recognition is the dot-product attention mechanism, originally developed for machine translation. The dot-product attention mechanism is a soft attention mechanism, which considers all positions when calculating the alignment matrix, assigning different weights to different positions.

[0003] In machine translation, the transcription of characters in language one to characters in language two does not correspond exactly to each other, so the alignment method based on soft attention can achieve good results.

[0004] However, the applicant discovered that for speech recognition tasks, the alignment between speech and characters is strictly monotonic, meaning that pronunciations correspond to characters in a sequential, one-to-one relationship. Because soft attention-based speech recognition models cannot explicitly learn this monotonic alignment, they suffer from low training efficiency and an inability to fully learn from the training data, leading to reduced model performance. This demonstrates the inherent limitations of traditional model training methods, including low training efficiency and poor model performance. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to propose a model training method, device, computer equipment and storage medium for speech recognition, so as to solve the problems of low training efficiency and low model performance in traditional model training methods.

[0006] In order to solve the above technical problems, the present application provides a model training method for speech recognition, which adopts the following technical solutions:

[0007] Acquire a model training sample, wherein the model training sample includes training audio data and text annotation data corresponding to the training audio;

[0008] input the training audio data into a speech recognition model to be trained to perform a speech recognition operation, to obtain a training recognition result;

[0009] construct an identification loss function corresponding to the speech recognition model to be trained according to the training recognition result and the text annotation data;

[0010] perform an audio feature extraction operation on the training audio data to obtain audio feature data;

[0011] perform a text feature recognition operation on the text annotation data to obtain text feature data;

[0012] construct an IMV loss function according to the audio feature data and the text feature data;

[0013] construct a joint loss function according to the identification loss function and the IMV loss function;

[0014] adjust parameters of the speech recognition model according to the joint loss function until the joint loss function meets a preset loss threshold, and then complete a model training operation on the speech recognition model to be trained.

[0015] To solve the above technical problems, the embodiment of the application also provides a model training device applied to speech recognition, which adopts the following technical scheme:

[0016] a sample acquisition module configured to acquire a model training sample, wherein the model training sample comprises training audio data and text annotation data corresponding to the training audio;

[0017] a speech recognition module configured to input the training audio data into a speech recognition model to be trained to perform a speech recognition operation, to obtain a training recognition result;

[0018] an identification loss construction module configured to construct an identification loss function corresponding to the speech recognition model to be trained according to the training recognition result and the text annotation data;

[0019] an audio feature extraction module configured to perform an audio feature extraction operation on the training audio data to obtain audio feature data;

[0020] a text feature recognition module configured to perform a text feature recognition operation on the text annotation data to obtain text feature data;

[0021] an IMV loss construction module configured to construct an IMV loss function according to the audio feature data and the text feature data;

[0022] A joint loss construction module, configured to construct a joint loss function based on the recognition loss function and the IMV loss function;

[0023] A parameter adjustment module is used to adjust the parameters of the speech recognition model according to the joint loss function until the joint loss function meets the preset loss threshold, thereby completing the model training operation of the speech recognition model to be trained.

[0024] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0025] It includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps of the model training method applied to speech recognition as described above.

[0026] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0027] The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the model training method for speech recognition as described above.

[0028] The present application provides a model training method for speech recognition, comprising: obtaining a model training sample, wherein the model training sample includes training audio data and text annotation data corresponding to the training audio; inputting the training audio data into the speech recognition model to be trained to perform a speech recognition operation to obtain a training recognition result; constructing a recognition loss function corresponding to the speech recognition model to be trained based on the training recognition result and the text annotation data; performing an audio feature extraction operation on the training audio data to obtain audio feature data; performing a text feature recognition operation on the text annotation data to obtain text feature data; constructing an IMV loss function based on the audio feature data and the text feature data; constructing a joint loss function based on the recognition loss function and the IMV loss function; adjusting the parameters of the speech recognition model based on the joint loss function until the joint loss function meets a preset loss threshold, thereby completing the model training operation on the speech recognition model to be trained. Compared with the existing technology, the present application combines an IMV-based loss function in the joint loss function, which can drive the model to explicitly and quickly learn the alignment relationship between speech and characters, thereby speeding up the training speed. Among them, the alignment constraint based on the IMV-based loss function does not require changing the model structure, is easy to implement, has a small amount of code engineering, and is suitable for any model that explicitly or implicitly uses the attention mechanism and requires monotonic alignment, such as speech recognition, speech synthesis, etc., and has universality and transferability; at the same time, because of the more accurate monotonic alignment relationship, the model can learn better representation capabilities under limited data, thereby effectively improving the recognition accuracy of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the solutions in this application, a brief introduction will be given below to the drawings required for use in the description of the embodiments of this application. Obviously, the drawings described below are some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0030] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0031] Figure 2 This is a flowchart of the implementation of the model training method for speech recognition provided in Example 1 of the present application;

[0032] Figure 3 yes Figure 2 A flowchart of a specific implementation of step S206;

[0033] Figure 4 This is a flowchart of calculating the attention alignment matrix A provided in Example 1 of the present application;

[0034] Figure 5 yes Figure 2 A flowchart of a specific implementation of step S207;

[0035] Figure 6 yes Figure 2 A flowchart of a specific implementation of step S204;

[0036] Figure 7 This is a flowchart of a specific implementation of the method for obtaining a Mel filter bank provided in Example 1 of the present application;

[0037] Figure 8 Schematic diagram of the structure of a model training device for speech recognition provided in Example 2 of the present application;

[0038] Figure 9 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0039] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0040] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0041] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0042] like Figure 1As shown, the system architecture 100 can include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is a medium for providing communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 can include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0043] A user can use the terminal devices 101, 102, 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 101, 102, 103, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0044] The terminal devices 101, 102, 103 can be various electronic devices with display screens and supporting web browsing, including but not limited to smartphones, tablet computers, e-book readers, MP3 (Moving Picture Experts Group Audio Layer III) players, MP4 (Moving Picture Experts Group Audio Layer IV) players, laptop computers, desktop computers, etc.

[0045] The server 105 can be a server providing various services, such as a background server supporting the pages displayed on the terminal devices 101, 102, 103.

[0046] It should be noted that the model training method for speech recognition provided by the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the model training device for speech recognition is generally provided in a server / terminal device.

[0047] It should be understood that Figure 1 The number of terminal devices, networks, and servers in

[0048] Embodiment One

[0049] With reference to Figure 2 , an implementation flowchart of the model training method for speech recognition provided by the embodiments of the present application is shown, and only the parts related to the present application are shown for ease of illustration.

[0050] The model training method for speech recognition described above includes the following steps:

[0051] Step S201: Obtain model training samples, wherein the model training samples include training audio data and text annotation data corresponding to the training audio.

[0052] In the embodiment of the present application, the model training samples are mainly used to train the end-to-end model so that the end-to-end model can perform automatic speech recognition.

[0053] In an embodiment of the present application, the training audio data refers to audio with higher clarity, which helps the end-to-end model to effectively extract audio feature information from the audio.

[0054] In the embodiment of the present application, the text annotation data refers to the text annotation information of the corresponding translation text pre-annotated by the user for the above-mentioned training audio data.

[0055] Step S202: input the training audio data into the speech recognition model to be trained to perform speech recognition operation and obtain a training recognition result.

[0056] Step S203: constructing a recognition loss function corresponding to the speech recognition model to be trained based on the training recognition results and the text annotation data.

[0057] Step S204: performing an audio feature extraction operation on the training audio data to obtain audio feature data.

[0058] In an embodiment of the present application, the audio feature extraction operation may be to convert the time domain of the training audio data into the frequency domain according to the fast Fourier transform, and filter the converted training audio data according to the Mel filter group to obtain the audio feature data.

[0059] Step S205: performing a text feature recognition operation on the text annotation data to obtain text feature data.

[0060] In an embodiment of the present application, the text feature recognition operation refers to extracting the feature vector that best reflects the characteristics of the character from the text annotation data, wherein the text feature extraction operation can be implemented by adopting a pixel-by-pixel feature extraction method, a skeleton feature extraction method, a vertical data statistical feature extraction method, a 13-point feature extraction method, an arc gradient feature extraction method, etc. Specifically, the pixel-by-pixel feature extraction method: the image is scanned row by row and column by column, the black pixel feature value is 1, the white pixel feature value is 0, and finally a feature vector matrix with the same dimension as the number of pixels in the image is formed. It should be understood that the examples of text feature extraction operations here are only for convenience of understanding and are not used to limit the present application.

[0061] Step S206: Construct an IMV loss function based on the audio feature data and the text feature data.

[0062] In an embodiment of the present application, the attention alignment matrix A of the audio feature data and the text feature data can be calculated according to the attention alignment algorithm, and then the index mapping vector π can be constructed according to the attention alignment matrix A, and finally the IMV loss function can be constructed according to the index mapping vector π.

[0063] Step S207: construct a joint loss function based on the recognition loss function and the IMV loss function.

[0064] In an embodiment of the present application, the recognition loss weight λ1 corresponding to the recognition loss function and the IMV loss weight λ2 corresponding to the IMV loss can be obtained first, and then the joint loss function can be constructed based on the recognition loss function, the IMV loss function, the recognition loss weight λ1 and the IMV loss weight λ2.

[0065] Step S208: Adjust the parameters of the speech recognition model according to the joint loss function until the joint loss function meets the preset loss threshold, and the model training operation of the speech recognition model to be trained is completed.

[0066] In an embodiment of the present application, a model training method for speech recognition is provided, including: obtaining a model training sample, wherein the model training sample includes training audio data and text annotation data corresponding to the training audio; inputting the training audio data into the speech recognition model to be trained to perform a speech recognition operation to obtain a training recognition result; constructing a recognition loss function corresponding to the speech recognition model to be trained based on the training recognition result and the text annotation data; performing an audio feature extraction operation on the training audio data to obtain audio feature data; performing a text feature recognition operation on the text annotation data to obtain text feature data; constructing an IMV loss function based on the audio feature data and the text feature data; constructing a joint loss function based on the recognition loss function and the IMV loss function; adjusting the parameters of the speech recognition model based on the joint loss function until the joint loss function meets a preset loss threshold, thereby completing the model training operation for the speech recognition model to be trained. Compared with the existing technology, the present application combines an IMV-based loss function in the joint loss function, which can drive the model to explicitly and quickly learn the alignment relationship between speech and characters, thereby speeding up the training speed. Among them, the alignment constraint based on the IMV-based loss function does not require changing the model structure, is easy to implement, has a small amount of code engineering, and is suitable for any model that explicitly or implicitly uses the attention mechanism and requires monotonic alignment, such as speech recognition, speech synthesis, etc., and has universality and transferability; at the same time, because of the more accurate monotonic alignment relationship, the model can learn better representation capabilities under limited data, thereby effectively improving the recognition accuracy of the model.

[0067] Continue reading Figure 3 , showing Figure 2A flowchart of a specific implementation of step S206 is shown, for ease of explanation, only the part related to this application is shown.

[0068] In some optional implementations of this embodiment, step S206 specifically includes:

[0069] Step S301: Calculate the attention alignment matrix A of the audio feature data and the text feature data according to the attention alignment algorithm.

[0070] In the embodiment of the present application, the attention alignment matrix A is the Dot-Product Attention alignment matrix A, wherein the Dot-Product Attention calculation method is shown in formula (1), and the calculation process is as follows: Figure 4 As shown in the figure, Q and K represent the two matrices that need to be aligned. For speech recognition, Q is the hidden feature matrix of the characters; K is the hidden feature matrix of the speech; MatMul refers to the dot multiplication operation of the matrix; Scale is the dimensional scaling; Mask is the mask operation; SoftMax is the normalization method, and the calculation method is shown in formula (2).

[0071]

[0072]

[0073] In the embodiment of the present application, A is the 1attention alignment matrix of Q and K. A is expressed as formula (3), where M is the hidden feature length of the input string, N is the hidden feature length of the current input speech, is a vector.

[0074] A=[α0,α1,...,α M ] (3)

[0075] Step S302: Construct an index mapping vector π according to the attention alignment matrix A, where the index mapping vector π is expressed as:

[0076] π=A T ·p=[α0·p, α1·p,...,α M ·p]

[0077] Where A represents the attention alignment matrix; p represents the number of the hidden state frame of the audio feature data; α M Represents the alignment vector of each character in the attention alignment matrix A; M represents the hidden feature length of the input string.

[0078] In the embodiment of the present application, the Index Mapping Vector (IMV) is a mathematical quantity proposed by the present invention for estimating the Attention alignment position. For the convenience of expression, it is denoted as π, and its calculation is shown in Formula (4).

[0079] π=A T ·p=[α0·p, α1·p,...,α M ·p] (4)

[0080] Where p is the number of the speech hidden state frame, p = [1, 2, 3, ..., N] T , A is the initial Attention alignment matrix calculated in step 1, π m It can be generally understood as the expected position of the mth character of the current input string aligned to the current input audio.

[0081] Step S303: Construct an IMV loss function based on the index mapping vector π, where the IMV loss function is expressed as:

[0082] Loss IMV =γ1{|Δπ-1|-(Δπ-1)}+γ2{|Δπ-K|+(Δπ-K)}

[0083] Among them, γ1 and γ2 represent weight coefficients respectively; Δπ=π[1:]-π[:-1].

[0084] In the embodiment of the present application, π m It represents the expected position of the mth character of the current input string aligned to the current input audio. Based on the assumption of short-term speech stability in speech recognition, a frame of speech is aligned to at most one character, and one character corresponds to at least one pronunciation. It can be concluded that in a string and audio parallel pair, as the character sequence number m increases, the expected alignment position of the character in the audio is π m On the other hand, based on physical facts, a character corresponds to at most K pronunciations, where K is a finite constant that can be set based on empirical values ​​or evaluated based on training data.

[0085] In some optional implementations of this embodiment, the attention alignment matrix is ​​expressed as:

[0086]

[0087] Among them, M represents the hidden feature length of the input character string; N represents the hidden feature length of the current input speech.

[0088] Continue reading Figure 5 , showing Figure 2A flowchart of a specific implementation of step S207 is shown, for ease of explanation, only the part related to this application is shown.

[0089] In some optional implementations of this embodiment, step S207 specifically includes:

[0090] Step S501: Obtain the recognition loss weight λ1 corresponding to the recognition loss function and the IMV loss weight λ2 corresponding to the IMV loss.

[0091] Step S502: Construct a joint loss function based on the recognition loss function, the IMV loss function, the recognition loss weight λ1, and the IMV loss weight λ2, wherein the joint loss function is expressed as:

[0092] Loss total =λ1·Loss ASR +λ2·Loss iMV .

[0093] Continue reading Figure 6 , showing Figure 2 A flowchart of a specific implementation of step S204 is shown, for ease of explanation, only the part related to this application is shown.

[0094] In some optional implementations of this embodiment, step S204 specifically includes:

[0095] Step S601: Convert the time domain of the training audio data into the frequency domain according to the Fast Fourier Transform.

[0096] Step S602: Filter the converted training audio data according to the Mel filter bank to obtain audio feature data.

[0097] In an embodiment of the present application, after obtaining the audio information, the audio information is preprocessed to achieve the purpose of enhancing the performance of the voice signal, and then the audio information is subjected to a fast Fourier transform to convert the audio information from the time domain to the frequency domain, and the audio information in the frequency domain is filtered by a Mel filter group whose frequency is set based on the language information of the audio information, thereby obtaining an audio feature vector of the audio information.

[0098] Continue reading Figure 7 , shows a flowchart of a specific implementation of the Mel filter group acquisition method provided in Example 1 of the present application. For the sake of convenience, only the part related to the present application is shown.

[0099] In some optional implementations of this embodiment, before step S602, the following steps are further included:

[0100] Step S701: Obtain training language information corresponding to the training audio data.

[0101] Step S702: Call a preset number of call filters corresponding to the training language information for sequential arrangement to obtain an initial mel filter bank.

[0102] Step S703: Determine the start mel frequency and the end mel frequency of each call filter in the initial mel filter bank to obtain a mel filter bank.

[0103] In the embodiments of the present application, after obtaining the audio information, the language of the audio information is analyzed while the preprocessing of the audio information is performed, so as to determine the language information corresponding to the audio information, i.e., the current audio information is output by which language, English or Chinese, or Japanese, etc.

[0104] In the embodiments of the present application, after obtaining the language information, since the audio pronunciation of different language information has different emphasis, after obtaining the language information, the initial frequency and the end frequency of each mel filter in the mel filter bank are set based on the characteristics of the language information, so that the language characteristics of the audio information can be highlighted after the current audio information passes through the mel filter bank with the frequency set based on the characteristics of the language information, so as to achieve the purpose of obtaining the audio feature vector of the audio information based on the characteristics of the language information, thereby achieving the purpose of accurately identifying the audio information.

[0105] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by computer readable instructions instructing related hardware, and the computer readable instructions can be stored in a computer readable storage medium. When the computer readable instructions are executed, the processes of the above-mentioned embodiments can be included. The storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0106] It should be understood that although each step in the flowchart of the accompanying drawings is displayed in sequence according to the arrow, these steps are not necessarily executed in sequence according to the arrow. Unless otherwise specified herein, the execution of these steps has no strict sequence limitation, and they can be executed in other orders. Moreover, at least part of the steps in the flowchart of the accompanying drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order is not necessarily sequential, but can be alternately executed with other steps or sub-steps or stages of other steps.

[0107] Example 2

[0108] Further references Figure 8 , as a response to the above Figure 2 The present application provides an embodiment of a model training device for speech recognition, which is similar to the embodiment of the present invention. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0109] like Figure 8 As shown, the model training device 200 for speech recognition of this embodiment includes: a sample acquisition module 210, a speech recognition module 220, a recognition loss construction module 230, an audio feature extraction module 240, a text feature recognition module 250, an IMV loss construction module 260, a joint loss construction module 270, and a parameter adjustment module 280. Among them:

[0110] A sample acquisition module 210 is configured to acquire a model training sample, wherein the model training sample includes training audio data and text annotation data corresponding to the training audio;

[0111] The speech recognition module 220 is used to input the training audio data into the speech recognition model to be trained to perform speech recognition operations and obtain training recognition results;

[0112] A recognition loss construction module 230 is used to construct a recognition loss function corresponding to the speech recognition model to be trained based on the training recognition results and the text annotation data;

[0113] An audio feature extraction module 240 is configured to perform an audio feature extraction operation on the training audio data to obtain audio feature data;

[0114] The text feature recognition module 250 is used to perform text feature recognition operations on the text annotation data to obtain text feature data;

[0115] An IMV loss construction module 260 is used to construct an IMV loss function based on the audio feature data and the text feature data;

[0116] A joint loss construction module 270 is used to construct a joint loss function based on the recognition loss function and the IMV loss function;

[0117] The parameter adjustment module 280 is used to adjust the parameters of the speech recognition model according to the joint loss function until the joint loss function meets the preset loss threshold, thereby completing the model training operation of the speech recognition model to be trained.

[0118] In the embodiment of the present application, the model training samples are mainly used to train the end-to-end model so that the end-to-end model can perform automatic speech recognition.

[0119] In an embodiment of the present application, the training audio data refers to audio with higher clarity, which helps the end-to-end model to effectively extract audio feature information from the audio.

[0120] In the embodiment of the present application, the text annotation data refers to the text annotation information of the corresponding translation text pre-annotated by the user for the above-mentioned training audio data.

[0121] In an embodiment of the present application, the audio feature extraction operation may be to convert the time domain of the training audio data into the frequency domain according to the fast Fourier transform, and filter the converted training audio data according to the Mel filter group to obtain the audio feature data.

[0122] In an embodiment of the present application, the text feature recognition operation refers to extracting the feature vector that best reflects the characteristics of the character from the text annotation data, wherein the text feature extraction operation can be implemented by adopting a pixel-by-pixel feature extraction method, a skeleton feature extraction method, a vertical data statistical feature extraction method, a 13-point feature extraction method, an arc gradient feature extraction method, etc. Specifically, the pixel-by-pixel feature extraction method: the image is scanned row by row and column by column, the black pixel feature value is 1, the white pixel feature value is 0, and finally a feature vector matrix with the same dimension as the number of pixels in the image is formed. It should be understood that the examples of text feature extraction operations here are only for convenience of understanding and are not used to limit the present application.

[0123] In an embodiment of the present application, the attention alignment matrix A of the audio feature data and the text feature data can be calculated according to the attention alignment algorithm, and then the index mapping vector π can be constructed according to the attention alignment matrix A, and finally the IMV loss function can be constructed according to the index mapping vector π.

[0124] In an embodiment of the present application, the recognition loss weight λ1 corresponding to the recognition loss function and the IMV loss weight λ2 corresponding to the IMV loss can be obtained first, and then the joint loss function can be constructed based on the recognition loss function, the IMV loss function, the recognition loss weight λ1 and the IMV loss weight λ2.

[0125] In an embodiment of the present application, a model training device 200 for speech recognition is provided, comprising: a sample acquisition module 210 for acquiring model training samples, wherein the model training samples include training audio data and text annotation data corresponding to the training audio; a speech recognition module 220 for inputting the training audio data into the speech recognition model to be trained to perform speech recognition operations and obtain training recognition results; a recognition loss construction module 230 for constructing a recognition loss function corresponding to the speech recognition model to be trained based on the training recognition results and the text annotation data; an audio feature extraction module 240 for extracting the training audio data; and a recognition loss function for extracting the training audio data. An audio feature extraction operation is performed on the audio feature data to obtain audio feature data; a text feature recognition module 250 is used to perform a text feature recognition operation on the text annotation data to obtain text feature data; an IMV loss construction module 260 is used to construct an IMV loss function based on the audio feature data and the text feature data; a joint loss construction module 270 is used to construct a joint loss function based on the recognition loss function and the IMV loss function; a parameter adjustment module 280 is used to adjust the parameters of the speech recognition model according to the joint loss function until the joint loss function meets the preset loss threshold, thereby completing the model training operation of the speech recognition model to be trained. Compared with the existing technology, the present application combines an IMV-based loss function in the joint loss function, which can drive the model to explicitly and quickly learn the alignment relationship between speech and characters, thereby speeding up the training speed. Among them, the alignment constraint based on the IMV-based loss function does not require changing the model structure, is easy to implement, has a small amount of code engineering, and is suitable for any model that explicitly or implicitly uses the attention mechanism and requires monotonic alignment, such as speech recognition, speech synthesis, etc., and has universality and transferability; at the same time, because of the more accurate monotonic alignment relationship, the model can learn better representation capabilities under limited data, thereby effectively improving the recognition accuracy of the model.

[0126] In some optional implementations of this embodiment, the IMV loss construction module 260 includes:

[0127] An alignment matrix calculation submodule is used to calculate the attention alignment matrix A of the audio feature data and the text feature data according to the attention alignment algorithm;

[0128] The mapping vector construction submodule is used to construct the index mapping vector π according to the attention alignment matrix A, where the index mapping vector π is expressed as:

[0129] π=A T ·p=[α0·p, α1·p,...,α M ·p]

[0130] Where A represents the attention alignment matrix; p represents the number of the hidden state frame of the audio feature data; αM represents the alignment vector of each character in the attention alignment matrix A; M represents the hidden feature length of the input string;

[0131] The IMV loss construction submodule is used to construct the IMV loss function according to the index mapping vector π, where the IMV loss function is expressed as:

[0132] Loss IMV =γ1{|Δπ-1|-(Δπ-1)}+γ2{|Δπ-K|+(Δπ-K)}

[0133] Among them, γ1 and γ2 represent weight coefficients respectively; Δπ=π[1:]-π[:-1].

[0134] In the embodiment of the present application, the attention alignment matrix A is the Dot-Product Attention alignment matrix A, wherein the Dot-Product Attention calculation method is shown in formula (1), and the calculation process is as follows: Figure 4 As shown in the figure, Q and K represent the two matrices that need to be aligned. For speech recognition, Q is the hidden feature matrix of the characters; K is the hidden feature matrix of the speech; MatMul refers to the dot multiplication operation of the matrix; Scale is the dimensional scaling; Mask is the mask operation; SoftMax is the normalization method, and the calculation method is shown in formula (2).

[0135]

[0136]

[0137] In the embodiment of the present application, A is the 1attention alignment matrix of Q and K. A is expressed as formula (3), where M is the hidden feature length of the input string, N is the hidden feature length of the current input speech, is a vector.

[0138] A=[α0,α1,...,α M ] (3)

[0139] In the embodiment of the present application, the Index Mapping Vector (IMV) is a mathematical quantity proposed by the present invention for estimating the Attention alignment position. For the convenience of expression, it is denoted as π, and its calculation is shown in Formula (4).

[0140] π=A T ·p=[α0·p, α1·p,...,α M ·p] (4)

[0141] Where p is the number of the speech hidden state frame, p = [1, 2, 3, ..., N] T , A is the initial Attention alignment matrix calculated in step 1, π m It can be generally understood as the expected position of the mth character of the current input string aligned to the current input audio.

[0142] In the embodiment of the present application, π m It represents the expected position of the mth character of the current input string aligned to the current input audio. Based on the assumption of short-term speech stability in speech recognition, a frame of speech is aligned to at most one character, and one character corresponds to at least one pronunciation. It can be concluded that in a string and audio parallel pair, as the character sequence number m increases, the expected alignment position of the character in the audio is π m On the other hand, based on physical facts, a character corresponds to at most K pronunciations, where K is a finite constant that can be set based on empirical values ​​or evaluated based on training data.

[0143] In some optional implementations of this embodiment, the attention alignment matrix is ​​expressed as:

[0144]

[0145] Among them, M represents the hidden feature length of the input character string; N represents the hidden feature length of the current input speech.

[0146] In some optional implementations of this embodiment, the joint loss construction module 270 includes:

[0147] The loss weight acquisition submodule is used to obtain the recognition loss weight λ1 corresponding to the recognition loss function and the IMV loss weight λ2 corresponding to the IMV loss;

[0148] The joint loss construction submodule is used to construct a joint loss function based on the recognition loss function, the IMV loss function, the recognition loss weight λ1, and the IMV loss weight λ2. The joint loss function is expressed as:

[0149] Loss total =λ1·Loss ASR +λ2·Loss IMV .

[0150] In some optional implementations of this embodiment, the audio feature extraction module 240 includes: a domain conversion submodule and a filtering processing submodule, wherein:

[0151] A domain conversion submodule for converting the time domain of the training audio data into the frequency domain based on the Fast Fourier Transform;

[0152] The filtering processing submodule is used to filter the converted training audio data according to the Mel filter bank to obtain audio feature data.

[0153] In an embodiment of the present application, after obtaining the audio information, the audio information is preprocessed to achieve the purpose of enhancing the performance of the voice signal, and then the audio information is subjected to a fast Fourier transform to convert the audio information from the time domain to the frequency domain, and the audio information in the frequency domain is filtered by a Mel filter group whose frequency is set based on the language information of the audio information, thereby obtaining an audio feature vector of the audio information.

[0154] In some optional implementations of this embodiment, the audio feature extraction module 240 further includes: a language information acquisition submodule, a sequence arrangement submodule, and a frequency determination submodule, wherein:

[0155] A language information acquisition submodule is used to obtain training language information corresponding to the training audio data;

[0156] A sequential arrangement submodule is used to call a preset number of call filters corresponding to the training language information and arrange them in sequence to obtain an initial Mel filter group;

[0157] The frequency determination submodule is used to determine the starting Mel frequency and the ending Mel frequency of each call filter in the initial Mel filter group to obtain the Mel filter group.

[0158] In an embodiment of the present application, after obtaining the audio information, the audio information is pre-processed and the language of the audio information is analyzed to determine the language information corresponding to the audio information, that is, the language in which the current audio information is output, whether it is English, Chinese, or Japanese, etc.

[0159] In an embodiment of the present application, after obtaining the language information, since different language information has different emphasis on the audio sound response, after obtaining the language information, it is necessary to set the initial frequency and end frequency of each mel filter in the mel filter group based on the characteristics of the language information, so that the current audio information can highlight the language characteristics of the audio information after passing through the mel filter group whose frequency is set based on the characteristics of the language information, so as to achieve the purpose of obtaining the audio feature vector of the audio information based on the characteristics of the language information, thereby achieving the purpose of accurately identifying the audio information.

[0160] In some optional implementations of this embodiment, the frequency determination submodule includes: a mode determination unit, a start frequency determination unit, and an end frequency determination unit, wherein:

[0161] a mode determining unit, configured to determine a first mode and a second mode according to training language information;

[0162] a starting frequency determining unit, configured to determine a specific starting Mel frequency of the k-th mel filter and the mel filters preceding it in the mel filter group according to a first method, and to determine a specific starting Mel frequency of the k+1-th mel filter and the mel filters following it in the mel filter group according to a second method, wherein k is a positive integer less than half the sum of a preset number and 1, and k+1 is a positive integer greater than or equal to half the sum of the preset number and 1;

[0163] an end frequency determination unit, configured to determine the specific end Mel frequency of the k-1th mel filter and the mel filter before it in the mel filter group according to a first method, and to determine the specific end Mel frequency of the kth mel filter and the mel filter after it in the mel filter group according to a second method, wherein the specific end Mel frequency of each mel filter is the specific start Mel frequency of the mel filter next to the mel filter.

[0164] To solve the above technical problems, the present application also provides a computer device. Figure 9 , Figure 9 This is a basic structural block diagram of the computer device in this embodiment.

[0165] The computer device 300 includes a memory 310, a processor 320, and a network interface 330 that are interconnected through a system bus. It should be noted that the figure only shows the computer device 300 having components 310-330, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0166] The computer device may be a desktop computer, notebook computer, PDA, cloud server, etc. The computer device may interact with the user via a keyboard, mouse, remote control, touchpad, or voice control device.

[0167] The memory 310 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, magnetic disk, optical disk, etc. In some embodiments, the memory 310 may be an internal storage unit of the computer device 300, such as a hard disk or memory of the computer device 300. In other embodiments, the memory 310 may also be an external storage device of the computer device 300, such as a plug-in hard disk equipped on the computer device 300, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Of course, the memory 310 may also include both the internal storage unit of the computer device 300 and its external storage device. In this embodiment, the memory 310 is generally used to store an operating system and various application software installed on the computer device 300, such as computer-readable instructions for a model training method for speech recognition. In addition, the memory 310 can also be used to temporarily store various data that has been output or is about to be output.

[0168] In some embodiments, the processor 320 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 320 is generally used to control the overall operation of the computer device 300. In this embodiment, the processor 320 is used to execute computer-readable instructions or process data stored in the memory 310, such as executing computer-readable instructions for the model training method for speech recognition.

[0169] The network interface 330 may include a wireless network interface or a wired network interface. The network interface 330 is generally used to establish a communication connection between the computer device 300 and other electronic devices.

[0170] The computer device provided in this application combines an IMV-based loss function in a joint loss function, which can drive the model to explicitly and quickly learn the alignment relationship between speech and characters, thereby accelerating the training speed. Among them, the alignment constraint based on the IMV-based loss function does not require changing the model structure, is easy to implement, has a small amount of code engineering, and is applicable to any model that explicitly or implicitly uses an attention mechanism and requires monotonic alignment, such as speech recognition, speech synthesis, etc., and has universality and transferability; at the same time, because of the more accurate monotonic alignment relationship, the model can learn better representation capabilities under limited data, thereby effectively improving the recognition accuracy of the model.

[0171] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the model training method applied to speech recognition as described above.

[0172] The computer-readable storage medium provided in this application combines an IMV-based loss function in a joint loss function, which can drive the model to explicitly and quickly learn the alignment relationship between speech and characters, thereby accelerating the training speed. Among them, the alignment constraint based on the IMV-based loss function does not require changing the model structure, is easy to implement, has a small amount of code engineering, and is applicable to any model that explicitly or implicitly uses an attention mechanism and requires monotonic alignment, such as speech recognition, speech synthesis, etc., and has universality and transferability; at the same time, because of the more accurate monotonic alignment relationship, the model can learn better representation capabilities under limited data, thereby effectively improving the recognition accuracy of the model.

[0173] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0174] Obviously, the embodiments described above are only some of the embodiments of the present application, rather than all of the embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned specific embodiments, or to make equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present application.

Claims

1. A model training method for speech recognition, characterized in that: The steps include: Acquire a model training sample, wherein the model training sample includes training audio data and text annotation data corresponding to the training audio; Inputting the training audio data into the speech recognition model to be trained to perform speech recognition operation to obtain a training recognition result; Constructing a recognition loss function corresponding to the speech recognition model to be trained according to the training recognition result and the text annotation data; Performing an audio feature extraction operation on the training audio data to obtain audio feature data; Performing a text feature recognition operation on the text annotation data to obtain text feature data; Constructing an IMV loss function based on the audio feature data and the text feature data; Constructing a joint loss function according to the recognition loss function and the IMV loss function; The parameters of the speech recognition model are adjusted according to the joint loss function until the joint loss function meets the preset loss threshold, thereby completing the model training operation of the speech recognition model to be trained.

2. The model training method for speech recognition according to claim 1, characterized in that: The step of constructing an IMV loss function based on the audio feature data and the text feature data specifically includes the following steps: Calculate the attention alignment matrix A of the audio feature data and the text feature data according to the attention alignment algorithm; An index mapping vector π is constructed according to the attention alignment matrix A, wherein the index mapping vector π is expressed as: π=A T ·p=[α0·p,α1·p,…,α M ·p] Wherein, A represents the attention alignment matrix; p represents the number of the hidden state frame of the audio feature data; α M represents the alignment vector of each character in the attention alignment matrix A; M represents the hidden feature length of the input string; The IMV loss function is constructed according to the index mapping vector π, wherein the IMV loss function is expressed as: Loss IMV =γ1{|Δπ-1|-(Δπ-1)}+γ2{|Δπ-K|+(Δπ-K)} Wherein, γ1 and γ2 represent weight coefficients respectively; Δπ=π[1:]-π[:-1]; K represents a finite constant.

3. The model training method for speech recognition according to claim 2, characterized in that: The attention alignment matrix is ​​expressed as: Among them, M represents the hidden feature length of the input string; N represents the hidden feature length of the current input speech; Z0, Z1, Z M Represent the 0th, 1st, and Mth elements of the input vector respectively.

4. The model training method for speech recognition according to claim 2, characterized in that: The step of constructing a joint loss function based on the recognition loss function and the IMV loss function specifically includes the following steps: Obtaining an identification loss weight λ1 corresponding to the identification loss function and an IMV loss weight λ2 corresponding to the IMV loss; The joint loss function is constructed according to the recognition loss function, the IMV loss function, the recognition loss weight λ1 and the IMV loss weight λ2, wherein the joint loss function is expressed as: Loss total =λ1·Loss ASR +λ2·Loss IMV 。 5. The model training method for speech recognition according to claim 1, characterized in that: The step of performing an audio feature extraction operation on the training audio data to obtain audio feature data specifically includes the following steps: Converting the time domain of the training audio data into the frequency domain according to a fast Fourier transform; The converted training audio data is filtered according to the Mel filter bank to obtain the audio feature data.

6. The model training method for speech recognition according to claim 5, characterized in that: Before the step of filtering the converted training audio data according to the Mel filter bank to obtain the audio feature data, the following steps are also included: Acquiring training language information corresponding to the training audio data; Calling a preset number of call filters corresponding to the training language information and arranging them in sequence to obtain an initial Mel filter group; The starting Mel frequency and the ending Mel frequency of each of the called filters in the initial Mel filter group are determined to obtain the Mel filter group.

7. The model training method for speech recognition according to claim 6, characterized in that: The step of determining the starting Mel frequency and the ending Mel frequency of each of the call filters in the initial Mel filter group to obtain the Mel filter group specifically includes the following steps: Determining a first method and a second method according to the training language information; Determining specific starting Mel frequencies for the kth mel filter and the mel filters preceding it in the mel filter group according to the first method, and determining specific starting Mel frequencies for the k+1th mel filter and the mel filters following it in the mel filter group according to the second method, where k is a positive integer less than half the sum of a preset number and 1, and k+1 is a positive integer greater than or equal to half the sum of the preset number and 1; Determine the specific termination Mel frequencies of the k-1th Mel filter and the Mel filters before it in the Mel filter group according to the first method, and determine the specific termination Mel frequencies of the kth Mel filter and the Mel filters after it in the Mel filter group according to the second method, wherein the specific termination Mel frequency of each Mel filter is the specific starting Mel frequency of the next Mel filter of the Mel filter.

8. A model training device for speech recognition, characterized in that: include: A sample acquisition module is used to acquire model training samples, wherein the model training samples include training audio data and text annotation data corresponding to the training audio; A speech recognition module is used to input the training audio data into the speech recognition model to be trained to perform speech recognition operations and obtain training recognition results; A recognition loss construction module, configured to construct a recognition loss function corresponding to the speech recognition model to be trained based on the training recognition result and the text annotation data; An audio feature extraction module, configured to perform an audio feature extraction operation on the training audio data to obtain audio feature data; A text feature recognition module, configured to perform a text feature recognition operation on the text annotation data to obtain text feature data; An IMV loss construction module, configured to construct an IMV loss function based on the audio feature data and the text feature data; A joint loss construction module, configured to construct a joint loss function based on the recognition loss function and the IMV loss function; A parameter adjustment module is used to adjust the parameters of the speech recognition model according to the joint loss function until the joint loss function meets the preset loss threshold, thereby completing the model training operation of the speech recognition model to be trained.

9. A computer device, characterized in that: It includes a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, it implements the steps of the model training method applied to speech recognition as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the model training method for speech recognition as described in any one of claims 1 to 7.

Images