Speech recognition model training method and device, computer device, and storage medium

Abstract

The application relates to an artificial intelligence technology and discloses a training method of a speech recognition model. The speech recognition model provided by the application comprises a decoder, the decoder comprises a multi-head cross attention unit, the multi-head cross attention unit comprises a Gaussian enhanced monotonic alignment sub-module, and the training method provided by the application comprises the following steps: sequentially processing audio data through a convolution sampling layer and an encoder to obtain a first audio encoding vector; processing text data through a vector mapping layer to obtain a first text encoding vector; obtaining a first attention matrix according to the first audio encoding vector and the first text encoding vector through a multi-head cross attention unit; reconstructing the first attention matrix through a Gaussian enhanced monotonic alignment sub-module to obtain a Gaussian enhanced monotonic alignment attention matrix; analyzing the Gaussian enhanced monotonic alignment attention matrix through a decoder to obtain a recognition result; and repeating the foregoing steps until all loss functions respectively reach convergence, so that a target speech recognition model is obtained.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a speech recognition model training method and device and related equipment. BACKGROUND

[0002] In the field of artificial intelligence technology, a traditional speech recognition model usually includes an acoustic model (AM), a pronunciation lexicon, and a language model (LM). The acoustic model, pronunciation lexicon, and language model all need to be learned and trained separately. An end-to-end speech recognition mechanism can enable the speech recognition model to discard the pronunciation lexicon and language model and truly convert speech into text.

[0003] Among them, there are two implementation methods of the end-to-end speech recognition mechanism: a CTC (Connectionist Temporal Classification) model and an encoder-decoder model based on an attention mechanism. However, the speech recognition accuracy of the CTC model and the encoder-decoder model based on the attention mechanism is not high, and the CTC model usually needs a language model to assist in re-scoring. SUMMARY

[0004] Embodiments of the present application provide a speech recognition model training method, device, computer equipment, and storage medium to solve the problem of low speech recognition accuracy of a traditional end-to-end speech recognition model.

[0005] In a first aspect of the present application, a speech recognition model training method is provided, the speech recognition model including a convolutional sampling layer, an encoder, a decoder, and a vector mapping layer. The decoder includes a multi-head cross-attention unit, and the multi-head cross-attention unit includes a Gaussian-enhanced monotonic alignment sub-module. The training method includes:

[0006] Inputting audio data in training sample data into the convolutional sampling layer for processing to obtain first audio feature sequence data;

[0007] Inputting the first audio feature sequence data into the encoder for processing to obtain a first audio encoding vector and calculating a connection time sequence classification loss of the encoder;

[0008] Inputting text data in the training sample data into the vector mapping layer for processing to obtain a first text encoding vector;

[0009] inputting the first audio encoding vector and the first text encoding vector into the multi-head cross attention unit for calculation to obtain a first attention matrix;

[0010] reconstructing the first attention matrix through the Gaussian enhanced monotonic alignment sub-module to obtain a Gaussian enhanced monotonic alignment attention matrix, and calculating a soft monotonic alignment loss of the Gaussian enhanced monotonic alignment sub-module;

[0011] analyzing the Gaussian enhanced monotonic alignment attention matrix through the decoder to obtain a speech recognition relationship between audio data and text data in the training sample data, and calculating a cross-entropy loss of the decoder;

[0012] repeating the foregoing steps until the connection time sequence classification loss, the soft monotonic alignment loss and the cross-entropy loss respectively reach convergence, to obtain a target speech recognition model.

[0013] In a second aspect of the present application, a speech recognition model training device is provided, the speech recognition model comprising a convolutional sampling layer, an encoder, a decoder, a vector mapping layer, the decoder comprising a multi-head cross attention unit, the multi-head cross attention unit comprising a Gaussian enhanced monotonic alignment sub-module, and the training device comprising:

[0014] a first data processing module configured to input audio data in training sample data into the convolutional sampling layer for processing to obtain a first audio feature sequence data;

[0015] a connection time sequence classification loss module configured to input the first audio feature sequence data into the encoder for processing to obtain a first audio encoding vector, and calculate a connection time sequence classification loss of the encoder;

[0016] a second data processing module configured to input text data in the training sample data into the vector mapping layer for processing to obtain a first text encoding vector;

[0017] a third data processing module configured to input the first audio encoding vector and the first text encoding vector into the multi-head cross attention unit for calculation to obtain a first attention matrix;

[0018] a soft monotonic alignment loss module configured to reconstruct the first attention matrix through the Gaussian enhanced monotonic alignment sub-module to obtain a Gaussian enhanced monotonic alignment attention matrix, and calculate a soft monotonic alignment loss of the Gaussian enhanced monotonic alignment sub-module;

[0019] a cross-entropy loss module configured to analyze the Gaussian enhanced monotonic alignment attention matrix through the decoder to obtain a speech recognition relationship between audio data and text data in the training sample data, and calculate a cross-entropy loss of the decoder;

[0020] a target speech recognition model module, configured to repeat the foregoing steps until the connection time sequence classification loss, the soft monotonic alignment loss and the cross-entropy loss respectively reach convergence, to obtain a target speech recognition model.

[0021] In a third aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the speech recognition model training method when executing the computer program.

[0022] In a fourth aspect, the present application provides a computer readable storage medium, which stores a computer program, wherein the computer program is executed by a processor to implement the steps of the speech recognition model training method.

[0023] The speech recognition model training method, device, computer device and storage medium, by the convolutional sampling layer and the encoder, process the audio data in sequence to obtain a first audio encoding vector, and calculate the connection time sequence classification loss of the encoder; by the vector mapping layer, process the text data to obtain a first text encoding vector; by the multi-head cross-attention unit, obtain a first attention matrix according to the first audio encoding vector and the first text encoding vector; by the Gaussian enhanced monotonic alignment sub-module, reconstruct the first attention matrix to obtain a Gaussian enhanced monotonic alignment attention matrix, and calculate the soft monotonic alignment loss of the Gaussian enhanced monotonic alignment sub-module; by the decoder, analyze the Gaussian enhanced monotonic alignment attention matrix to obtain a recognition result, and calculate the cross-entropy loss of the decoder; repeat the foregoing steps until all losses respectively reach convergence, to obtain a target speech recognition model. Not only the speech recognition accuracy of the traditional end-to-end speech recognition model is improved, but also the computing performance of the traditional end-to-end speech recognition model is improved. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0025] Figure 1 is an application environment diagram of the speech recognition model training method in an embodiment of the present application;

[0026] Figure 2 is a flowchart of the speech recognition model training method in an embodiment of the present application;

[0027] Figure 3is a structural schematic diagram of a speech recognition model training device in an embodiment of the present application.

[0028] Figure 4 is a flowchart of a speech recognition method in an embodiment of the present application.

[0029] Figure 5 is a structural schematic diagram of a speech recognition device in an embodiment of the present application.

[0030] Figure 6 is a schematic diagram of a computer device in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of the present application.

[0032] The speech recognition model training method provided by the present application can be applied in an application environment such as Figure 1 The computer device can be, but is not limited to, various personal computers, notebook computers, and the computer device can also be a server. The server can be a standalone server or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms. It can be understood that Figure 1 The number of computer devices in the application environment is only illustrative, and can be expanded in any number according to actual needs.

[0033] The embodiments of the present application can acquire and process related data based on artificial intelligence technology. Artificial intelligence (AI) is the use of digital computers or computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, robot technology, biometric technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0034] In an embodiment, the speech recognition model training method comprises the following steps. Figure 2As shown, a speech recognition model training method is provided, the speech recognition model comprising a convolutional sampling layer, an encoder, a decoder, a vector mapping layer, the decoder comprising a multi-head cross-attention unit, the multi-head cross-attention unit comprising a Gaussian enhanced monotonic alignment sub-module, and the method is applied to Figure 1 The computer device in the method is taken as an example for illustration, comprising the following steps S101 to S107:

[0035] S101, input audio data in training sample data into the convolutional sampling layer for processing to obtain first audio feature sequence data.

[0036] The convolutional sampling layer is used to extract various target acoustic features of audio from the audio data in the training sample data and abstract the target acoustic features, which not only greatly reduces the training parameters of the speech recognition model, but also reduces the overfitting degree of the speech recognition model. It should be specially noted that the specific structure design and operation mode of the convolutional sampling layer are not described herein again as the convolutional sampling layer is not the core part of the present application.

[0037] S102, input the first audio feature sequence data into the encoder for processing to obtain first audio encoding vectors and calculate the connection time sequence classification loss of the encoder.

[0038] The encoder is based on the encoder in the Transformer, which converts the received first audio feature sequence data into the first audio encoding vectors and calculates the connection time sequence classification loss in the foregoing conversion process. Further, the encoder uses the CTC algorithm in the foregoing conversion process, which is used to solve the problem of inconsistent lengths of input series and output series in the technical field of speech recognition, text recognition, etc., i.e., the problem of inconsistent lengths of the first audio feature sequence data and the first audio encoding vectors in the present application. Further, the calculation result of the CTC algorithm is used as the connection time sequence classification loss (CTC Loss). The CTC algorithm is added in the training process of the speech recognition model, which realizes that the training only needs an input sequence and an output sequence without the need for alignment processing of training data in advance, and the CTC directly outputs the probability of sequence prediction without the need for external post-processing.

[0039] S103, input the text data in the training sample data into the vector mapping layer for processing to obtain a first text encoding vector.

[0040] The vector mapping layer (Embedding) extracts various target text features of the text from the text data in the sample data, abstracts the target text features, and converts the text data into the first text encoding vector. Specifically, the processing techniques used by the vector mapping layer include but are not limited to word2vec, fasttext, glove, character embedding, elmo, bert, etc. It should be specially pointed out that the specific structure design and operation mode of the vector mapping layer are not described here because it is not the core part of the present application.

[0041] S104, input the first audio encoding vector and the first text encoding vector into the multi-head cross attention unit for calculation to obtain a first attention matrix.

[0042] Further, the decoder not only includes the multi-head cross attention unit (Multi Head Cross Attention, MHCA), but also includes a multi-head self-attention unit (Multi Head Self Attention, MHSA) and a feed forward layer (Feed Forward). Further, before the first audio encoding vector and the first text encoding vector are input into the multi-head cross attention unit for calculation, the multi-head self-attention unit is used to process the first text encoding vector to extract more rich text feature information. Further, after the first attention matrix is obtained, the first attention matrix is sent to the feed forward layer for processing. It should be specially pointed out that the multi-head self-attention unit and the feed forward layer are not the core part of the present application, so the specific structure design and operation principle of the multi-head attention unit and the feed forward layer are not described here.

[0043] S105, reconstruct the first attention matrix through the Gaussian enhanced monotone alignment sub-module to obtain a Gaussian enhanced monotone alignment attention matrix, and calculate a soft monotone alignment loss of the Gaussian enhanced monotone alignment sub-module.

[0044] Further, the reconstructing the first attention matrix by the Gaussian enhanced monotonic alignment module to obtain a Gaussian enhanced monotonic alignment attention matrix comprises: first, constructing a first sequence mapping vector by the first attention matrix, and setting a first constraint condition of the first sequence mapping vector, wherein the first constraint condition contains a first difference value between a current character output and a previous character output. Then, inputting the first difference value into a first activation function to obtain a second difference value, wherein the second difference value is always greater than or equal to 0. Finally, accumulating the second difference value to obtain a second sequence mapping vector, and using the second sequence mapping vector to construct the Gaussian enhanced monotonic alignment attention matrix.

[0045] Further, the first sequence mapping vector is constructed by the following formula:

[0046]

[0047] wherein T1 represents a length of an input audio feature sequence, a represents the first attention matrix, p represents an input sequence [0, 1, 2, …, T1-1], π i represents the first sequence mapping vector.

[0048] Further, the first constraint condition is:

[0049]

[0050] wherein Δπ i represents a difference value between a current sequence mapping vector and a previous sequence mapping vector, T2 represents a length of an output character feature sequence, and D represents a first hyperparameter constant. In another more specific embodiment, the first hyperparameter constant is set as 50 by default. The first constraint condition ensures that each output character corresponds to 1 to D frames of input data.

[0051] Further, the first activation function is a linear rectifier function, and the second difference value is calculated by the following formula:

[0052]

[0053] wherein ReLU represents a linear rectifier function, and Δπ′ i represents the second difference value. The linear rectifier function can ensure that the input frame corresponding to the current character output is always greater than or equal to the previous character.

[0054] Further, the second difference value is accumulated according to the following formula to obtain the second sequence mapping vector:

[0055]

[0056] wherein π′ irepresenting the second sequence mapping vector.

[0057] Further, the Gaussian enhanced monotonic alignment attention matrix is constructed according to the following formula:

[0058]

[0059] wherein, σ represents the standard deviation of the Gaussian kernel, and α' represents the first hyperparameter constant. i,j representing the Gaussian enhanced monotonic alignment attention matrix, and exp represents the exponential function with the natural constant e as the base.

[0060] Further, the soft monotonic alignment loss is calculated according to the following formula:

[0061] l SMA = λ1|||Δπ-1|-(Δπ-1)||1+ λ2|||Δπ-D|+(Δπ-D)||1

[0062] wherein, λ1 represents the second hyperparameter constant, and λ2 represents the third hyperparameter constant. In another more specific embodiment, the second hyperparameter constant is set to 0.8, and the third hyperparameter constant is set to 0.2.

[0063] S106, parsing the Gaussian enhanced monotonic alignment attention matrix through the decoder to obtain the speech recognition relationship between the audio data and the text data in the training sample data, and calculating the cross-entropy loss of the decoder.

[0064] Further, the obtained speech recognition relationship between the audio data and the text data is artificially verified to obtain an artificial verification result. Then, the artificial verification result is sent to an associated model designer to provide data support for the model designer to artificially intervene and modify the parameters of the speech recognition model.

[0065] S107, repeating the foregoing steps until the connection time sequence classification loss, the soft monotonic alignment loss and the cross-entropy loss respectively reach convergence to obtain a target speech recognition model.

[0066] Further, before the repeating the foregoing steps until the connection time sequence classification loss, the soft monotonic alignment loss and the cross-entropy loss respectively reach convergence, it further includes: optimizing each parameter of the encoder according to the connection time sequence classification loss, optimizing each parameter of the Gaussian enhanced monotonic alignment sub-module according to the soft monotonic alignment loss, and optimizing each parameter of the decoder according to the cross-entropy loss.

[0067] Further, the first audio encoding vector, the first text encoding vector and the target speech recognition model are uploaded to a blockchain. Specifically, corresponding summary information is obtained based on the first audio encoding vector, the first text encoding vector and the speech recognition model respectively, and specifically, the summary information is obtained by hashing the first audio encoding vector, the first text encoding vector and the target speech recognition model, for example, by using a sha256s algorithm. Uploading the summary information to the blockchain can ensure its security and fairness and transparency to the user. The user device can download the summary information from the blockchain to verify whether the first audio encoding vector, the first text encoding vector and the target speech recognition model are tampered with. The blockchain referred to in this example is a new application mode of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism and encryption algorithm. Blockchain, in essence, is a decentralized database, a string of data blocks associated using cryptographic methods, each data block containing information about a batch of network transactions, used to verify the validity of the information (anti-fake) and generate the next block. The blockchain can include a blockchain underlying platform, a platform product service layer and an application service layer, etc.

[0068] The speech recognition model training method, device, computer device and storage medium provided in this embodiment obtain a first audio encoding vector by sequentially processing audio data through a convolutional sampling layer and an encoder, and calculate a connection time sequence classification loss of the encoder; obtain a first text encoding vector by processing text data through a vector mapping layer; obtain a first attention matrix according to the first audio encoding vector and the first text encoding vector through a multi-head cross attention unit; obtain a Gaussian-enhanced monotonic alignment attention matrix by reconstructing the first attention matrix through a Gaussian-enhanced monotonic alignment sub-module, and calculate a soft monotonic alignment loss of the Gaussian-enhanced monotonic alignment sub-module; obtain a recognition result by analyzing the Gaussian-enhanced monotonic alignment attention matrix through a decoder, and calculate a cross-entropy loss of the decoder; repeat the foregoing steps until all losses respectively reach convergence, and obtain a target speech recognition model. Not only the speech recognition accuracy of the traditional end-to-end speech recognition model is improved, but also the computing performance of the traditional end-to-end speech recognition model is improved.

[0069] Figure 4 is a flowchart of a speech recognition method using a speech recognition model trained according to the speech recognition model training method described above, as shown in Figure 4 The speech recognition method includes the following steps S201 to S204:

[0070] S201, input the speech data to be recognized into the convolutional sampling layer in the speech recognition model for processing to obtain a second audio feature sequence data.

[0071] S202, input the second audio feature sequence data to an encoder in the speech recognition model for processing to obtain a second audio encoding vector.

[0072] S203, input the second audio encoding vector to a decoder in the speech recognition model for processing to obtain a second text encoding vector.

[0073] S204, input the second text encoding vector to a vector mapping layer in the speech recognition model for processing to obtain a target audio recognition text result.

[0074] Exemplarily, an insurance customer service system of a certain financial technology service platform trains the foregoing speech recognition model according to the foregoing speech recognition model training method through collected training data, and applies the trained speech recognition model to the insurance customer service system. In the process of communication between a customer service staff of the insurance customer service system and an insurance user using a target insurance product App, if the customer service staff or the insurance user inputs a segment of voice data and enables the speech recognition function added by the target insurance product App, the segment of voice data is sent to the speech recognition model, the speech recognition model outputs a text form of a speech recognition result of the segment of voice data, and the text form of the speech recognition result is returned to the customer service staff or the insurance user.

[0075] Further, first, the voice data to be identified and the target audio recognition text result are stored. Then, the voice data to be identified and the target audio recognition text result are sent to the corresponding voice recognition request initiator, and the accuracy rate data feedback of the voice recognition request initiator to the target audio recognition text result is received. Finally, on the one hand, the accuracy rate data feedback is used to optimize the parameters of the voice recognition model and the aforementioned training steps of the voice recognition model, and on the other hand, a voice recognition accuracy threshold range is set, the mathematical statistical value of the accuracy rate data is calculated by a mathematical statistical method, if the mathematical statistical value of the accuracy rate data exceeds the voice recognition accuracy threshold range, voice recognition early warning information containing the mathematical statistical value of the accuracy rate data and the voice recognition accuracy threshold range is generated, and the voice recognition early warning information is sent to the relevant technical personnel, prompting the relevant technical personnel to optimize the voice recognition model and the training process of the voice recognition model to obtain a voice recognition model with higher recognition accuracy for replacement. For example, when the technical personnel of a certain financial service platform receives the emergency early warning information that the voice recognition accuracy of the voice recognition model in the insurance customer service system of the platform has decreased more than the preset voice recognition accuracy threshold range, the technical personnel of the platform will quickly optimize the parameters of the voice recognition model and the training process of the voice recognition model according to the content contained in the emergency early warning information, so as to obtain a new voice recognition model with online use label voice recognition accuracy.

[0076] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0077] In an embodiment, a voice recognition model training device 100 is provided, which corresponds to the voice recognition model training method in the above embodiment. The voice recognition model includes a convolutional sampling layer, an encoder, a decoder, and a vector mapping layer. The decoder includes a multi-head cross-attention unit, and the multi-head cross-attention unit includes a Gaussian enhanced monotonic alignment sub-module. As shown in the figure, the voice recognition model training device 100 includes a first data processing module, a connection time sequence classification loss module 12, a second data processing module 13, a third data processing module 14, a soft monotonic alignment loss module 15, a cross-entropy loss module 16, and a target voice recognition model module 17. The functions of each module are described in detail as follows: Figure 3

[0078] The first data processing module 11 is configured to input the audio data in the training sample data into the convolutional sampling layer for processing to obtain first audio feature sequence data.

[0079] ​The connection time sequence classification loss module 12 is configured to input the first audio feature sequence data into the encoder for processing to obtain a first audio encoding vector and calculate a connection time sequence classification loss of the encoder.

[0080] The second data processing module 13 is configured to input text data in the training sample data into the vector mapping layer for processing to obtain a first text encoding vector.

[0081] The third data processing module 14 is configured to input the first audio encoding vector and the first text encoding vector into the multi-head cross attention unit for calculation to obtain a first attention matrix.

[0082] The soft monotonic alignment loss module 15 is configured to reconstruct the first attention matrix through the Gaussian enhanced monotonic alignment submodule to obtain a Gaussian enhanced monotonic alignment attention matrix and calculate a soft monotonic alignment loss of the Gaussian enhanced monotonic alignment submodule.

[0083] The cross-entropy loss module 16 is configured to parse the Gaussian enhanced monotonic alignment attention matrix through the decoder to obtain a speech recognition relationship between audio data and text data in the training sample data and calculate a cross-entropy loss of the decoder.

[0084] The target speech recognition model module 17 is configured to repeat the foregoing steps until the connection time sequence classification loss, the soft monotonic alignment loss and the cross-entropy loss respectively reach convergence to obtain a target speech recognition model.

[0085] Further, the soft monotonic alignment loss module 15 further comprises:

[0086] The first sequence mapping vector submodule is configured to construct a first sequence mapping vector through the first attention matrix and set a first restriction condition of the first sequence mapping vector, wherein the first restriction condition comprises a first difference value between a current character output and a previous character output.

[0087] The first activation function submodule is configured to input the first difference value into a first activation function to obtain a second difference value, wherein the second difference value is always greater than or equal to 0.

[0088] The second sequence mapping vector submodule is configured to accumulate the second difference value to obtain a second sequence mapping vector and use the second sequence mapping vector to construct the Gaussian enhanced monotonic alignment attention matrix.

[0089] Further, the first sequence mapping vector submodule further comprises:

[0090] The first formula submodule is configured to construct the first sequence mapping vector through the following formula:

[0091]

[0092] wherein, T1 represents the length of the input audio feature sequence, a represents the first attention matrix, p represents the input sequence [0, 1, 2, …, T1-1], π i represents the first sequence mapping vector.

[0093] The first restriction condition subunit is configured to:

[0094]

[0095] wherein, Δπ i represents the difference between the current sequence mapping vector and the previous sequence mapping vector, T2 represents the length of the output character feature sequence, and D represents a hyperparameter constant.

[0096] Further, the first activation function sub-module further includes:

[0097] The linear rectifier function subunit is configured to: when the first activation function is a linear rectifier function, the second difference value is calculated by the following formula:

[0098]

[0099] wherein, ReLU represents a linear rectifier function, Δπ′ i represents the second difference value.

[0100] Further, the second sequence mapping vector sub-module further includes:

[0101] The second formula subunit is configured to: according to the following formula, the second difference value is accumulated to obtain the second sequence mapping vector:

[0102]

[0103] wherein, π′ i represents the second sequence mapping vector.

[0104] The attention matrix subunit is configured to: according to the following formula, the Gaussian-enhanced monotonic alignment attention matrix is constructed:

[0105]

[0106] wherein, σ represents the standard deviation of the Gaussian kernel, a′ i,j represents the Gaussian-enhanced monotonic alignment attention matrix.

[0107] Figure 5 FIG. 2 is a structural schematic diagram of a speech recognition device 200 according to an embodiment of the present application, as shown in Figure 5As shown, the speech recognition device 200 provided by the speech recognition model training device 100 for speech recognition includes a second audio feature sequence module 21, a second audio encoding vector module 22, a second text encoding vector module 23, and an audio recognition text result module 24. The functions of each module are described in detail as follows:

[0108] The second audio feature sequence module 21 is configured to input speech data to be recognized into a convolutional sampling layer in the speech recognition model for processing to obtain second audio feature sequence data.

[0109] The second audio encoding vector module 22 is configured to input the second audio feature sequence data into an encoder in the speech recognition model for processing to obtain a second audio encoding vector.

[0110] The second text encoding vector module 23 is configured to input the second audio encoding vector into a decoder in the speech recognition model for processing to obtain a second text encoding vector.

[0111] The audio recognition text result module 24 is configured to input the second text encoding vector into a vector mapping layer in the speech recognition model for processing to obtain a target audio recognition text result.

[0112] The "first" and "second" in the above modules / units are only used to distinguish different modules / units, and are not used to limit the priority of which module / unit is higher or other limiting meanings. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or device that includes a series of steps or modules does not have to be limited to only those steps or modules clearly listed, but can include other steps or modules that are not clearly listed or inherent to these processes, methods, products, or devices. The division of modules in this application is only a logical division, and in actual application, there can be another division method.

[0113] For specific limitations of the speech recognition model training device, refer to the limitations of the speech recognition model training method in the above, which will not be repeated here. Each module in the speech recognition model training device can be realized by software, hardware, and combinations thereof, in whole or in part. Each module can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0114] In one embodiment, a computer device is provided, which can be a server, and its internal structure diagram can be as shown in Figure 6As shown in the figure. The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store the data involved in the speech recognition model training method. The network interface of the computer device is used to communicate with the external terminal through the network connection. The computer program is executed by the processor to implement a speech recognition model training method.

[0115] In one embodiment, a computer device is provided, including a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the steps of the speech recognition model training method in the above-mentioned embodiments, for example Figure 2 The steps S101 to S107 and other extensions and related steps of the method are shown. Alternatively, the processor executes the computer program to implement the functions of each module / unit of the speech recognition model training device in the above-mentioned embodiments, for example Figure 3 The functions of the modules 11 to 17 are shown. To avoid repetition, they will not be described here.

[0116] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The processor is the control center of the computer device, and connects all parts of the computer device through various interfaces and lines.

[0117] The memory can be used to store the computer program and / or modules, and the processor realizes various functions of the computer device by running or executing the computer program and / or modules stored in the memory, and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), and the like; and the data storage area can store data created according to the use of the mobile phone (such as audio data, video data, etc.), and the like.

[0118] The memory can be integrated in the processor or arranged separately from the processor.

[0119] In one embodiment, a computer readable storage medium is provided, and the computer program is stored on the computer readable storage medium. The computer program is executed by the processor to realize the steps of the voice recognition model training method in the above embodiment, for example Figure 2 The steps S101 to S107 and the extension of other extensions and related steps of the method are shown. Alternatively, the computer program is executed by the processor to realize the functions of each module / unit of the voice recognition model training device in the above embodiment, for example Figure 3 The functions of the modules 11 to 17 are shown. To avoid repetition, details are not described here.

[0120] A person of ordinary skill in the art can understand that all or part of the processes in the above embodiment methods can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium, and when executed, can include the processes of the above embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided in the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0121] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0122] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method of training a speech recognition model, the speech recognition model comprising a convolutional sampling layer, an encoder, a decoder, a vector mapping layer, the decoder comprising a multi-head cross-attention unit, characterized in that, The training method comprises: inputting audio data in training sample data into the convolutional sampling layer for processing to obtain first audio feature sequence data; inputting the first audio feature sequence data into the encoder for processing to obtain a first audio encoding vector, and calculating a connection time sequence classification loss of the encoder; inputting text data in the training sample data into the vector mapping layer for processing to obtain a first text encoding vector; inputting the first audio encoding vector and the first text encoding vector into the multi-head cross attention unit for calculation to obtain a first attention matrix; reconstructing the first attention matrix to obtain a Gaussian enhanced monotonic alignment attention matrix, and calculating a soft monotonic alignment loss; analyzing the Gaussian enhanced monotonic alignment attention matrix through the decoder to obtain a speech recognition relationship between the audio data and the text data in the training sample data, and calculating a cross-entropy loss of the decoder; repeating the foregoing steps until the connection time sequence classification loss, the soft monotonic alignment loss and the cross-entropy loss respectively reach convergence, to obtain a target speech recognition model; wherein the reconstructing the first attention matrix to obtain a Gaussian enhanced monotonic alignment attention matrix comprises: constructing a first sequence mapping vector through the first attention matrix, and setting a first restriction condition of the first sequence mapping vector, wherein the first restriction condition comprises a first difference value between a current text output and a previous text output; inputting the first difference value into a first activation function to obtain a second difference value, wherein the second difference value is always greater than or equal to 0; accumulating the second difference value to obtain a second sequence mapping vector, and using the second sequence mapping vector to construct the Gaussian enhanced monotonic alignment attention matrix wherein the first sequence mapping vector is constructed by the following formula: wherein, denotes a length of an input audio feature sequence, denotes the first attention matrix, denotes an input sequence , denotes the first sequence mapping vector; wherein the first restriction condition is: wherein, denotes the difference between the current sequence mapping vector and the previous sequence mapping vector, denotes the length of the output text feature sequence, denotes a hyperparameter constant; wherein the soft monotonic alignment loss is calculated according to the following formula: wherein, represents a second hyperparameter constant, represents a third hyperparameter constant. 2.The method of Claim 1, wherein, the first activation function is a linear rectifier function, and the second difference value is calculated by the following formula: wherein, represents a linear rectifier function, represents the second difference. 3.The method of Claim 2, wherein, the second difference value is accumulated according to the following formula to obtain the second sequence mapping vector: wherein denotes the second sequence mapping vector. 4.The method of Claim 3, wherein, the Gaussian enhanced monotonic alignment attention matrix is constructed according to the following formula: wherein, denotes a standard deviation of a Gaussian kernel, denotes the Gaussian enhanced monotonic alignment attention matrix. 5.A device for training a speech recognition model, the speech recognition model comprising a convolutional sampling layer, an encoder, a decoder, a vector mapping layer, the decoder comprising a multi-head cross-attention unit, characterized in that, The training device comprises: a first data processing module configured to input audio data in training sample data into the convolutional sampling layer for processing to obtain first audio feature sequence data; a connection time sequence classification loss module configured to input the first audio feature sequence data into the encoder for processing to obtain a first audio encoding vector, and calculate a connection time sequence classification loss of the encoder; a second data processing module configured to input text data in the training sample data into the vector mapping layer for processing to obtain a first text encoding vector; a third data processing module configured to input the first audio encoding vector and the first text encoding vector into the multi-head cross attention unit for calculation to obtain a first attention matrix; a soft monotonic alignment loss module configured to reconstruct the first attention matrix to obtain a Gaussian enhanced monotonic alignment attention matrix, and calculate a soft monotonic alignment loss; a cross-entropy loss module, configured to parse the Gaussian enhanced monotonic alignment attention matrix by the decoder to obtain a speech recognition relationship between audio data and text data in the training sample data, and calculate a cross-entropy loss of the decoder; a target speech recognition model module, configured to repeat the foregoing steps until the connection time sequence classification loss, the soft monotonic alignment loss and the cross-entropy loss respectively reach convergence, so as to obtain a target speech recognition model; wherein the soft monotonic alignment loss module further comprises: a first sequence mapping vector submodule, configured to construct a first sequence mapping vector by the first attention matrix, and set a first constraint condition of the first sequence mapping vector, wherein the first constraint condition comprises a first difference value between a current character output and a previous character output; a first activation function submodule, configured to input the first difference value into a first activation function to obtain a second difference value, wherein the second difference value is always greater than or equal to 0; and a second sequence mapping vector submodule, configured to accumulate the second difference value to obtain a second sequence mapping vector, and use the second sequence mapping vector to construct the Gaussian enhanced monotonic alignment attention matrix; wherein the first sequence mapping vector submodule further comprises a first formula subunit and a first constraint condition subunit, and the first formula subunit is configured to construct the first sequence mapping vector by the following formula: wherein, denotes a length of an input audio feature sequence, denotes the first attention matrix, denotes an input sequence , denotes the first sequence mapping vector; the first constraint subunit is configured to have the first constraint as: wherein, denotes the difference between the current sequence mapping vector and the previous sequence mapping vector, denotes the length of the output text feature sequence, denotes a hyperparameter constant; wherein the soft monotonic alignment loss is calculated according to the following formula: wherein, represents a second hyperparameter constant, represents a third hyperparameter constant.

6. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the speech recognition model training method according to any one of claims 1 to 4.

7. A computer-readable storage medium storing a computer program, wherein the computer program comprises the following steps of: The computer program is executed by the processor to implement the steps of the speech recognition model training method according to any one of claims 1 to 4.

Images