Speech recognition method, speech recognition model training method and device, and equipment

Abstract

The present disclosure provides a speech recognition method, a training method and device of a speech recognition model, and equipment, which can be applied to the fields of computer technology, speech recognition technology and finance. The speech recognition method comprises: performing feature extraction on to-be-recognized speech data to obtain a mel spectrum feature vector sequence, wherein the mel spectrum feature vector sequence comprises at least one position code and a mel spectrum feature vector corresponding to each position code; processing the mel spectrum feature vector sequence to obtain a sparse spectrum feature vector sequence, wherein the sparse spectrum feature vector sequence comprises at least one position code and a sparse spectrum feature vector corresponding to each position code; performing fusion processing on the mel spectrum feature vector sequence and the sparse spectrum feature vector sequence according to the at least one position code to obtain a fusion feature vector sequence; and processing the fusion feature vector sequence to obtain a speech recognition result.

Description

Technical Field

[0001] This disclosure relates to the fields of computer technology, speech recognition technology, and finance, and more specifically, to a speech recognition method and apparatus, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] With the development of computer technology, speech recognition technology has been widely applied in various scenarios. Speech recognition technology refers to the technology by which computers recognize and understand input speech signals and convert those signals into corresponding text or commands.

[0003] In realizing the present invention, the inventors discovered that the related technology has at least the following problems: due to the influence of background noise in noisy speech scenarios, the recognition rate of speech recognition is easily reduced, thus failing to guarantee the accuracy of speech recognition. Summary of the Invention

[0004] In view of the above, this disclosure provides a speech recognition method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

[0005] According to one aspect of this disclosure, a speech recognition method is provided, comprising:

[0006] Feature extraction is performed on the speech data to be recognized to obtain a Mel spectrum feature vector sequence, wherein the Mel spectrum feature vector sequence includes at least one position code and a Mel spectrum feature vector corresponding to each of the at least one position code;

[0007] The above Mel spectrum feature vector sequence is processed to obtain a sparse spectrum feature vector sequence, wherein the sparse spectrum feature vector sequence includes at least one position code and a sparse spectrum feature vector corresponding to each of the at least one position code.

[0008] Based on at least one of the aforementioned positional codes, the aforementioned Mel-spectral feature vector sequence and the aforementioned sparse-spectral feature vector sequence are fused to obtain a fused feature vector sequence; and

[0009] The above fused feature vector sequence is processed to obtain the speech recognition result.

[0010] According to embodiments of this disclosure, the above-mentioned fusion processing of the Mel-spectral feature vector sequence and the sparse spectral feature vector sequence based on at least one positional encoding to obtain a fused feature vector sequence includes:

[0011] For each of the at least one position code mentioned above,

[0012] Determine the target Mel-spectral feature vector and the target sparse-spectral feature vector corresponding to the above positional encoding;

[0013] Based on the aforementioned target Mel-spectral feature vectors, determine the first query matrix, the first key matrix, and the first value matrix;

[0014] Based on the aforementioned target sparse spectral feature vectors, determine the second query matrix, the second key matrix, and the second value matrix; and

[0015] Based on the first query matrix, the first key matrix, the first value matrix, the second query matrix, the second key matrix, and the second value matrix, the fusion feature vector corresponding to the position encoding is determined.

[0016] According to embodiments of this disclosure, determining the fused feature vector corresponding to the position code based on the first query matrix, the first key matrix, the first value matrix, the second query matrix, the second key matrix, and the second value matrix includes:

[0017] Based on the first query matrix and the second key matrix described above, determine the first attention weight matrix;

[0018] Based on the second query matrix and the first key matrix described above, determine the second attention weight matrix;

[0019] Based on the first attention weight matrix and the second value matrix mentioned above, the first feature vector is obtained;

[0020] Based on the second attention weight matrix and the first value matrix described above, the second feature vector is obtained; and

[0021] Based on the first feature vector and the second feature vector, the fused feature vector is obtained.

[0022] According to embodiments of this disclosure, the above-mentioned processing of the Mel spectrum feature vector sequence to obtain a sparse spectrum feature vector sequence includes:

[0023] Based on the above Mel spectrum feature vector sequence, determine the target dictionary;

[0024] For each of the at least one position code mentioned above,

[0025] Based on the aforementioned target dictionary, the Mel-spectral feature vectors corresponding to the aforementioned positional codes are processed to obtain the sparse spectral feature vectors corresponding to the aforementioned positional codes; and

[0026] The sparse spectral feature vector sequence is determined based on the sparse spectral feature vector corresponding to each of the above-mentioned positions.

[0027] According to embodiments of this disclosure, determining the target dictionary based on the aforementioned Mel spectrum feature vector sequence includes:

[0028] Construct a dictionary training sample set based on at least one of the aforementioned Mel spectrum feature vectors;

[0029] Based on the aforementioned dictionary training sample set, determine the initial dictionary; and

[0030] Based on the dictionary learning algorithm, the initial dictionary is processed to obtain the target dictionary.

[0031] According to embodiments of this disclosure, the above-mentioned processing of the Mel-spectral feature vector corresponding to the positional encoding based on the target dictionary to obtain the sparse spectral feature vector corresponding to the positional encoding includes:

[0032] Based on the tracking algorithm, sparse representation of the Mel-spectral feature vector is performed to obtain sparse coding coefficients; and

[0033] Based on the target dictionary and the sparse coding coefficients, the sparse spectral feature vector corresponding to the positional coding is determined.

[0034] According to embodiments of this disclosure, the above-mentioned feature extraction of the speech data to be recognized to obtain the Mel spectrum feature vector sequence includes:

[0035] The above-mentioned speech data to be recognized is preprocessed to obtain Mel spectrum sequence data, wherein the above-mentioned Mel spectrum sequence data includes at least one Mel spectrum data.

[0036] Based on the aforementioned Mel spectrum sequence data and position-coded sequence data, intermediate coded sequence data is determined, wherein the aforementioned position-coded sequence data includes at least one of the aforementioned position codes; and

[0037] Feature extraction is performed on the above intermediate encoded sequence data to obtain the above Mel spectrum feature vector sequence.

[0038] According to embodiments of this disclosure, the preprocessing of the aforementioned speech data to be recognized to obtain Mel spectrum sequence data includes:

[0039] The above speech data to be recognized is subjected to data augmentation processing to obtain augmented speech data;

[0040] The enhanced speech data is then windowed to obtain the target speech data; and

[0041] Feature extraction was performed on the target speech data to obtain the Mel spectrum sequence data.

[0042] According to embodiments of this disclosure, the above-described speech recognition method is implemented based on a deep learning model, which includes a feature extraction layer, an attention layer, and a temporal classification layer. The method further includes:

[0043] The above-mentioned speech data to be recognized is input into the above-mentioned feature extraction layer to obtain the above-mentioned Mel spectrum feature vector sequence and the above-mentioned sparse spectrum feature vector sequence;

[0044] The above Mel spectrum feature vector sequence and the above sparse spectrum feature vector sequence are input into the above attention layer to obtain the above fused feature vector sequence;

[0045] The above-mentioned fused feature vector sequence is input into the above-mentioned temporal classification layer to obtain the above-mentioned speech recognition result.

[0046] According to another aspect of this disclosure, a method for training a speech recognition model is provided, comprising:

[0047] Feature extraction is performed on at least one speech data sample to be identified to obtain a sample Mel spectrum feature vector sequence corresponding to each of the at least one speech data sample to be identified. The sample Mel spectrum feature vector sequence includes at least one sample position code and a sample Mel spectrum feature vector corresponding to each of the at least one sample position code. Each of the at least one speech data sample to be identified corresponds to a real speech recognition result.

[0048] The sample Mel spectrum feature vector sequence corresponding to each of the above-mentioned at least one sample speech data to be identified is processed to obtain the sample sparse spectrum feature vector sequence corresponding to each of the above-mentioned at least one sample speech data to be identified, wherein the sample sparse spectrum feature vector sequence includes the above-mentioned at least one sample position code and the sample sparse spectrum feature vector corresponding to each of the above-mentioned at least one sample position code.

[0049] Based on the at least one sample position encoding, the sample Mel-spectral feature vector sequence and the sample sparse-spectral feature vector sequence corresponding to each of the at least one unidentified sample speech data are respectively fused to obtain the sample fused feature vector sequence corresponding to each of the at least one unidentified sample speech data; and

[0050] Using the real speech recognition results corresponding to each of the above-mentioned at least one speech data to be recognized and the above-mentioned sample fusion feature vector sequence, a deep learning model is trained to obtain a speech recognition model.

[0051] According to embodiments of this disclosure, the deep learning model described above includes an attention layer.

[0052] According to embodiments of this disclosure, the above-mentioned fusion processing of the sample Mel-spectral feature vector sequence and the sample sparse spectral feature vector sequence corresponding to each of the at least one sample speech data to be identified, based on the at least one sample position encoding, to obtain the sample fusion feature vector sequence corresponding to each of the at least one sample speech data to be identified, includes:

[0053] The sample Mel spectrum feature vector sequence and the sample sparse spectrum feature vector sequence corresponding to each of the above-mentioned at least one sample speech data to be identified are input into the attention layer to obtain the sample fusion feature vector sequence corresponding to each of the above-mentioned at least one sample speech data to be identified.

[0054] According to embodiments of this disclosure, the above-mentioned use of the above-mentioned real speech recognition results corresponding to each of the above-mentioned at least one speech data to be recognized sample and the above-mentioned sample fusion feature vector sequence to train a deep learning model to obtain a speech recognition model includes:

[0055] The sample fusion feature vector sequence corresponding to each of the above-mentioned at least one sample speech data to be identified is input into the above-mentioned deep learning model to obtain the sample speech recognition result corresponding to each of the above-mentioned at least one sample speech data to be identified.

[0056] Based on the actual speech recognition results and the sample speech recognition results corresponding to each of the at least one speech data to be recognized, the loss function values ​​corresponding to each of the at least one speech data to be recognized are obtained; and

[0057] The model parameters of the deep learning model are adjusted according to the loss function values ​​corresponding to each of the at least one speech data sample to be identified, until the predetermined termination condition is met, thus obtaining the speech recognition model.

[0058] According to embodiments of this disclosure, the deep learning model further includes a temporal classification layer, which comprises a first one-dimensional convolutional layer, a second one-dimensional convolutional layer, a max pooling layer, a fully connected layer, and a normalization layer.

[0059] According to embodiments of this disclosure, the input of the sample fusion feature vector sequence corresponding to each of the at least one sample speech data to be identified into the deep learning model to obtain the sample speech recognition result corresponding to each of the at least one sample speech data to be identified includes:

[0060] For each of the sample fusion feature vector sequences corresponding to the at least one speech data sample to be identified,

[0061] The above sample fusion feature vector sequence is input into the first one-dimensional convolutional layer to obtain the first intermediate sample feature vector sequence.

[0062] The first intermediate sample feature vector sequence is input into the second one-dimensional convolutional layer to obtain the second intermediate sample feature vector sequence.

[0063] The second intermediate sample feature vector sequence is input into the max pooling layer to obtain the third intermediate sample feature vector sequence.

[0064] The third intermediate sample feature vector sequence is input into the fully connected layer to obtain the fourth intermediate sample feature vector sequence; and

[0065] The feature vector sequence of the fourth intermediate sample is input into the normalization layer to obtain the speech recognition result of the sample.

[0066] According to another aspect of this disclosure, a voice recognition device is provided, comprising:

[0067] The first feature extraction module is used to extract features from the speech data to be recognized to obtain a Mel spectrum feature vector sequence, wherein the Mel spectrum feature vector sequence includes at least one position code and a Mel spectrum feature vector corresponding to each of the at least one position code.

[0068] The first processing module is used to process the above-mentioned Mel spectrum feature vector sequence to obtain a sparse spectrum feature vector sequence, wherein the above-mentioned sparse spectrum feature vector sequence includes at least one position code and a sparse spectrum feature vector corresponding to each of the above-mentioned at least one position code.

[0069] The first fusion processing module is configured to fuse the Mel-spectral feature vector sequence and the sparse spectral feature vector sequence according to at least one positional encoding, to obtain a fused feature vector sequence; and

[0070] The second processing module is used to process the above-mentioned fused feature vector sequence to obtain the speech recognition result.

[0071] According to another aspect of this disclosure, a training apparatus for a speech recognition model is provided, comprising:

[0072] The second feature extraction module is used to extract features from at least one speech data sample to be identified, and obtain a sample Mel spectrum feature vector sequence corresponding to each of the at least one speech data sample to be identified. The sample Mel spectrum feature vector sequence includes at least one sample position code and a sample Mel spectrum feature vector corresponding to each of the at least one sample position code. Each of the at least one speech data sample to be identified corresponds to a real speech recognition result.

[0073] The third processing module is used to process the sample Mel spectrum feature vector sequence corresponding to each of the above-mentioned at least one sample speech data to be identified, to obtain the sample sparse spectrum feature vector sequence corresponding to each of the above-mentioned at least one sample speech data to be identified, wherein the sample sparse spectrum feature vector sequence includes the above-mentioned at least one sample position code and the sample sparse spectrum feature vector corresponding to each of the above-mentioned at least one sample position code.

[0074] The second fusion processing module is configured to fuse the sample Mel-spectral feature vector sequence and the sample sparse spectral feature vector sequence corresponding to each of the at least one sample speech data to be identified, based on the at least one sample position encoding, to obtain a sample fusion feature vector sequence corresponding to each of the at least one sample speech data to be identified; and

[0075] The training module is used to train a deep learning model using the real speech recognition results corresponding to each of the above-mentioned at least one speech data to be recognized sample and the above-mentioned sample fusion feature vector sequence, so as to obtain a speech recognition model.

[0076] According to another aspect of this disclosure, an electronic device is provided, comprising:

[0077] One or more processors;

[0078] Memory, used to store one or more instructions.

[0079] When one or more of the above instructions are executed by one or more processors, the one or more processors cause the one or more processors to implement the method as described in this disclosure.

[0080] According to another aspect of this disclosure, a computer-readable storage medium is provided having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the methods described in this disclosure.

[0081] According to another aspect of this disclosure, a computer program product is provided, which includes computer-executable instructions that, when executed, are used to perform the methods described in this disclosure.

[0082] According to embodiments of this disclosure, since the Mel-spectral feature vector sequence is obtained by feature extraction from the speech data to be recognized, it can be used to characterize the Mel-spectral features of the speech data to be recognized. Since the sparse spectral feature vector sequence is obtained by processing the Mel-spectral feature vector sequence, it can be used to characterize the sparse spectral features of the speech data to be recognized. Furthermore, by fusing the Mel-spectral feature vector sequence and the sparse spectral feature vector sequence, the Mel-spectral features and sparse spectral features can complement each other, thus at least partially overcoming the technical problem in related technologies where background noise in noisy speech scenes easily leads to low speech recognition rates, thereby improving the accuracy of speech recognition. Attached Figure Description

[0083] The above and other objects, features and advantages of this disclosure will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0084] Figure 1 The illustration schematically shows a system architecture for applying speech recognition methods and speech recognition model training methods according to embodiments of the present disclosure;

[0085] Figure 2 A flowchart illustrating a speech recognition method according to an embodiment of the present disclosure is shown schematically.

[0086] Figure 3 This illustration schematically shows an example of the process of extracting features from speech data to be recognized to obtain a sequence of Mel spectrum feature vectors according to an embodiment of the present disclosure;

[0087] Figure 4 This schematically illustrates a flowchart of a method for processing a Mel-spectral feature vector sequence to obtain a sparse spectral feature vector sequence according to an embodiment of the present disclosure;

[0088] Figure 5 This illustration schematically shows an example of a process for processing a Mel-spectral feature vector sequence to obtain a sparse spectral feature vector sequence according to an embodiment of the present disclosure.

[0089] Figure 6 This illustration schematically shows an example of a process of fusing a Mel-spectral feature vector sequence and a sparse-spectral feature vector sequence according to at least one positional encoding, according to an embodiment of the present disclosure, to obtain a fused feature vector sequence.

[0090] Figure 7 A flowchart illustrating a method for training a speech recognition model according to an embodiment of the present disclosure is shown schematically.

[0091] Figure 8The illustration shows an example diagram of the training process of a speech recognition model according to an embodiment of the present disclosure;

[0092] Figure 9 The illustration shows an example diagram of the training process of a speech recognition model according to another embodiment of the present disclosure;

[0093] Figure 10 A block diagram of a speech recognition device according to an embodiment of the present disclosure is shown schematically;

[0094] Figure 11 A block diagram schematically illustrates a training apparatus for a speech recognition model according to an embodiment of the present disclosure; and

[0095] Figure 12 A block diagram of an electronic device suitable for implementing a speech recognition method and a training method for a speech recognition model according to embodiments of the present disclosure is shown schematically. Detailed Implementation

[0096] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0097] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0098] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0099] When using expressions such as "at least one of A, B, and C," the expression should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.). Similarly, when using expressions such as "at least one of A, B, or C," the expression should generally be interpreted in accordance with the meaning commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, or C" should include, but is not limited to, systems having A alone, having B alone, having C alone, having A and B, having A and C, having B and C, and / or having A, B, and C, etc.).

[0100] In the technical solution disclosed herein, the acquisition, storage, and application of user personal information comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and there is no violation of public order and good morals.

[0101] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0102] In related technologies, speech recognition models are typically trained by combining features from two different domains, allowing the two features to complement each other. However, because this approach does not focus on denoising or enhancing the audio signal, if the original signal is contaminated with noise, both types of feature information extracted from the original signal may also be contaminated, making it impossible to supplement the lost information by combining the two features.

[0103] Furthermore, features from different modalities can be fused. For example, audio and video features can be mapped one-to-one during speech recognition model training, allowing the two modalities to complement each other. However, because this approach requires fusing features from different modalities, it results in a large and overly complex speech recognition model.

[0104] In summary, during the process of realizing the present invention, the inventors discovered at least the following problems in the related technology: due to the influence of background noise in noisy speech scenarios, the recognition rate of speech recognition is easily reduced, thus failing to guarantee the accuracy of speech recognition.

[0105] To at least partially address the technical problems existing in related technologies, this disclosure provides a speech recognition method, a training method and apparatus for a speech recognition model, and a device that can be applied to the fields of computer technology, speech recognition technology, and finance. The speech recognition method includes: extracting features from the speech data to be recognized to obtain a Mel-spectral feature vector sequence. The Mel-spectral feature vector sequence includes at least one positional code and a Mel-spectral feature vector corresponding to each of the at least one positional code. The Mel-spectral feature vector sequence is processed to obtain a sparse spectral feature vector sequence. The sparse spectral feature vector sequence includes at least one positional code and a sparse spectral feature vector corresponding to each of the at least one positional code. Based on the at least one positional code, the Mel-spectral feature vector sequence and the sparse spectral feature vector sequence are fused to obtain a fused feature vector sequence; the fused feature vector sequence is then processed to obtain a speech recognition result.

[0106] It should be noted that the speech recognition method and apparatus provided in this disclosure can be used in the fields of computer technology, speech recognition technology, and finance, such as in the field of internet technology. The speech recognition method and apparatus provided in this disclosure can also be used in any field other than computer technology, speech recognition technology, and finance, such as in the field of information technology. The application fields of the speech recognition method and apparatus provided in this disclosure are not limited.

[0107] Figure 1 The illustration schematically depicts a system architecture according to embodiments of the present disclosure where speech recognition methods and speech recognition model training methods can be applied. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0108] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0109] Users can interact with server 105 via network 104 using at least one of the first terminal device 101, second terminal device 102, and third terminal device 103 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, second terminal device 102, and third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0110] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0111] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0112] It should be noted that the speech recognition method provided in this embodiment can generally be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or it can be executed by other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103. Accordingly, the speech recognition device provided in this embodiment can also be disposed in the first terminal device 101, the second terminal device 102, or the third terminal device 103, or in other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0113] Alternatively, the speech recognition method provided in this embodiment can also be executed by server 105. Correspondingly, the speech recognition device provided in this embodiment can generally be located in server 105. The speech recognition method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the speech recognition device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0114] It should be noted that the speech recognition model training method provided in this embodiment can generally be executed by server 105. Correspondingly, the speech recognition model training device provided in this embodiment can generally be located in server 105. The speech recognition model training method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the speech recognition model training device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0115] Alternatively, the speech recognition model training method provided in this embodiment of the present disclosure can also be executed by the first terminal device 101, the second terminal device 102, or the third terminal device 103, or by other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103. Correspondingly, the speech recognition model training device provided in this embodiment of the present disclosure can also be located in the first terminal device 101, the second terminal device 102, or the third terminal device 103, or in other terminal devices different from the first terminal device 101, the second terminal device 102, or the third terminal device 103.

[0116] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0117] It should be noted that the sequence numbers of the operations in the following methods are for descriptive purposes only and should not be considered as indicating the execution order of the operations. Unless explicitly stated otherwise, the method does not need to be executed in the exact order shown.

[0118] Figure 2 A flowchart illustrating a speech recognition method according to an embodiment of the present disclosure is shown schematically.

[0119] like Figure 2 As shown, the method 200 includes operations S210 to S240.

[0120] In operation S210, feature extraction is performed on the speech data to be recognized to obtain a Mel spectrum feature vector sequence, wherein the Mel spectrum feature vector sequence includes at least one position code and a Mel spectrum feature vector corresponding to each of the at least one position code.

[0121] In operation S220, the Mel spectrum feature vector sequence is processed to obtain a sparse spectrum feature vector sequence, wherein the sparse spectrum feature vector sequence includes at least one position code and a sparse spectrum feature vector corresponding to each of the at least one position code.

[0122] In operation S230, the Mel spectrum feature vector sequence and the sparse spectrum feature vector sequence are fused according to at least one position code to obtain a fused feature vector sequence.

[0123] In operation S240, the fused feature vector sequence is processed to obtain the speech recognition result.

[0124] According to embodiments of this disclosure, the code for a speech recognition request can be pre-written into a script. In response to detecting a user's voice input operation, the server can run the script to generate a speech recognition request message. The server can then send this speech recognition request message to the client, enabling the client to perform speech recognition on the voice data to be recognized corresponding to the voice input operation based on the speech recognition request message.

[0125] According to embodiments of this disclosure, after receiving the speech data to be recognized, preprocessing can be performed on the speech data to obtain target speech data. The target speech data can be a speech data matrix to be recognized. The speech data matrix to be evaluated can include at least one frame of speech sub-data to be evaluated. Adjacent frames of speech sub-data to be evaluated may have common components. After obtaining the target speech data, a Fourier transform can be performed on the target speech data to obtain the speech data matrix to be evaluated in the frequency domain. Acoustic feature extraction can be performed on the speech data matrix to be evaluated in the frequency domain to obtain an acoustic feature sequence.

[0126] For example, acoustic features may include at least one of the following: Fbank (i.e., FilterBank), Mel-Frequency Cepstral Coefficients (MFCC), timbre vector, zero-crossing rate, subband energy, subband energy entropy, spectral center, spectral extension, spectral entropy, spectral flux, spectral roll-off, and timbre deviation.

[0127] According to embodiments of this disclosure, feature extraction of the speech data to be recognized to obtain a Mel spectrum feature vector sequence may include: extracting Mel frequency cepstral coefficient features from the speech data matrix to be evaluated in the frequency domain to obtain Mel spectrum sequence data. After obtaining the Mel spectrum sequence data, a positional encoding method can be used to encode the position of at least one Mel spectrum data included in the Mel spectrum sequence data to obtain a positional code corresponding to at least one Mel spectrum data. The positional encoding method can be set according to actual business needs and is not limited thereto. For example, the positional encoding method may include a sine / cosine positional encoding method or a learned position vector method.

[0128] For example, a positional encoding method can be used to encode the positions corresponding to each of the at least one Mel-spectrum data points included in the Mel-spectrum sequence data, resulting in positional codes corresponding to each of the at least one Mel-spectrum data points. Alternatively, a positional encoding method can be used to encode the positions corresponding to a subset of the at least one Mel-spectrum data points included in the Mel-spectrum sequence data, resulting in positional codes corresponding to a subset of the at least one Mel-spectrum data points.

[0129] According to embodiments of this disclosure, after obtaining the Mel spectrum eigenvector sequence, the Mel spectrum eigenvector sequence can be sparsely represented to obtain a sparse spectral eigenvector sequence. Sparse representation can refer to a method of expressing most or all of the original signal using fewer basis signals to obtain a concise representation of the signal.

[0130] For example, a target dictionary can be determined based on the Mel-spectral feature vector sequence. The target dictionary can refer to a set of atoms. An atom can refer to a basis signal. After obtaining the target dictionary, the Mel-spectral feature vectors corresponding to the positional codes can be processed to obtain sparse spectral feature vectors corresponding to the positional codes. Based on this, a sparse spectral feature vector sequence can be determined according to the sparse spectral feature vectors corresponding to each positional code.

[0131] According to embodiments of this disclosure, after obtaining the Mel-spectral feature vector sequence and the sparse-spectral feature vector sequence, the Mel-spectral feature vector and the sparse-spectral feature vector corresponding to each position code can be fused according to each position code in at least one position code to obtain a fused feature vector corresponding to the position code. A fused feature vector sequence is obtained based on the fused feature vector corresponding to each position code. The fusion processing method can be set according to actual business needs and is not limited here.

[0132] For example, the fusion processing method may include at least one of the following: feature concatenation, cross-modal attention, and conditional batch normalization (CBN).

[0133] According to embodiments of this disclosure, after obtaining the fused feature vector sequence, the fused feature vector sequence and the positional encoding sequence can be decoded to obtain recognition spectrum sequence data. After obtaining the recognition spectrum sequence data, the recognition spectrum sequence data can be processed to obtain the speech recognition result.

[0134] According to embodiments of this disclosure, since the Mel-spectral feature vector sequence is obtained by feature extraction from the speech data to be recognized, it can be used to characterize the Mel-spectral features of the speech data to be recognized. Since the sparse spectral feature vector sequence is obtained by processing the Mel-spectral feature vector sequence, it can be used to characterize the sparse spectral features of the speech data to be recognized. Furthermore, by fusing the Mel-spectral feature vector sequence and the sparse spectral feature vector sequence, the Mel-spectral features and sparse spectral features can complement each other, thus at least partially overcoming the technical problem in related technologies where background noise in noisy speech scenes easily leads to low speech recognition rates, thereby improving the accuracy of speech recognition.

[0135] The following is for reference. Figures 3-6 The speech recognition method 200 according to an embodiment of the present invention will be further described.

[0136] According to embodiments of this disclosure, operation S210 may include the following operations.

[0137] The speech data to be recognized is preprocessed to obtain Mel spectrum sequence data, wherein the Mel spectrum sequence data includes at least one Mel spectrum data. Based on the Mel spectrum sequence data and the positional encoding sequence data, intermediate encoding sequence data is determined, wherein the positional encoding sequence data includes at least one positional code. Feature extraction is performed on the intermediate encoding sequence data to obtain a Mel spectrum feature vector sequence.

[0138] According to embodiments of this disclosure, the preprocessing method can be set according to actual business needs, and is not limited thereto. For example, the preprocessing method may include at least one of the following: data augmentation processing, windowing processing, and acoustic feature extraction processing.

[0139] According to embodiments of this disclosure, after obtaining the Mel spectrum sequence data, a positional encoding method can be used to encode the position of at least one Mel spectrum data included in the Mel spectrum sequence data based on the positional encoding sequence data, thereby obtaining intermediate encoded sequence data. After obtaining the intermediate encoded sequence data, feature extraction can be performed on the intermediate encoded sequence data to obtain a Mel spectrum feature vector sequence.

[0140] According to embodiments of this disclosure, preprocessing the speech data to be recognized to obtain Mel spectrum sequence data may include the following operations.

[0141] Data augmentation is performed on the speech data to be recognized to obtain augmented speech data. Windowing is then applied to the augmented speech data to obtain the target speech data. Feature extraction is then performed on the target speech data to obtain Mel-spectral sequence data.

[0142] According to embodiments of this disclosure, in response to detecting a user's voice input operation, an audio signal can be acquired. The audio signal is then converted into speech data to be recognized. For example, the audio sampling rate can be set to 16 kHz. After acquiring the speech data to be recognized, due to the spectral tilt phenomenon in speech, data augmentation processing can be performed on the speech data to be recognized to obtain enhanced speech data with increased high-frequency speech energy.

[0143] According to embodiments of this disclosure, after obtaining the enhanced speech data, in order to ensure the smoothness of the acoustic feature parameters, an overlapping frame extraction method can be used to window the enhanced speech data to obtain the target speech data. For example, the frame length of each frame can be set to 20ms or 25ms, and the frame shift can be set to 10ms. In this case, there can be overlapping parts between adjacent frames.

[0144] For example, a discrete Fourier transform can be performed on enhanced speech data with a window length of 25ms to obtain one frame, and then the calculation can continue for the next frame by stepping 10ms, which is equivalent to adding a rectangular window. The specific form of the rectangular window can be set according to actual business needs and is not limited here. For example, the rectangular window can include at least one of the following: Hamming window and Hanning window.

[0145] According to embodiments of this disclosure, after obtaining target speech data, Mel filter banks can be used to process the target speech data to obtain Mel spectrum sequence data.

[0146] Figure 3 The illustration shows an example of a process for extracting features from speech data to be recognized and obtaining a sequence of Mel spectrum feature vectors according to an embodiment of the present disclosure.

[0147] like Figure 3 As shown in step 300, data augmentation processing can be performed on the speech data 301 to be recognized to obtain enhanced speech data 302. Windowing processing is then applied to the enhanced speech data 302 to obtain the target speech data 303. Feature extraction is then performed on the target speech data 303 to obtain Mel spectrum sequence data 304.

[0148] After obtaining the Mel spectrum sequence data 304, the intermediate coding sequence data 306 can be determined based on the Mel spectrum sequence data 304 and the position coding sequence data 305. Feature extraction is then performed on the intermediate coding sequence data 306 to obtain the Mel spectrum feature vector sequence 307.

[0149] Figure 4 The flowchart illustrates a method for processing a Mel spectrum feature vector sequence to obtain a sparse spectrum feature vector sequence according to an embodiment of the present disclosure.

[0150] like Figure 4 As shown, the method 400 includes operations S421 to S423.

[0151] In operation S421, the target dictionary is determined based on the Mel spectrum feature vector sequence.

[0152] In operation S422, for each position code in at least one position code, the Mel spectral feature vector corresponding to the position code is processed according to the target dictionary to obtain the sparse spectral feature vector corresponding to the position code.

[0153] In operation S423, a sparse spectral feature vector sequence is determined based on the sparse spectral feature vector corresponding to each position.

[0154] According to embodiments of this disclosure, operation S421 may include the following operations.

[0155] A dictionary training sample set is constructed based on at least one Mel-spectral feature vector. An initial dictionary is determined based on this training sample set. The initial dictionary is then processed using a dictionary learning algorithm to obtain the target dictionary.

[0156] According to embodiments of this disclosure, dictionary learning can refer to finding the optimal basis under sparse representation, which satisfies the constraint of coefficient conditional uniqueness while obtaining a more accurate and sparser representation. A dictionary learning algorithm may include a dictionary construction phase (i.e., DictionaryGenerate) and a sample representation phase using a dictionary (i.e., Sparsecoding with a precomputed dictionary).

[0157] According to embodiments of this disclosure, the dictionary training sample set can refer to a portion of the mel spectrum feature vectors from at least one mel spectrum feature vector. For example, features of all phonemes can be selected from at least one mel spectrum feature vector as the dictionary training sample set, such that the dictionary training sample set contains different feature classes of the mel spectrum feature vectors, to ensure a relatively consistent feature space distribution. A phoneme can refer to the smallest unit of speech defined based on the natural properties of speech.

[0158] According to embodiments of this disclosure, mel spectrum feature vectors for different phonemes can be selected from at least one mel spectrum feature vector. The selected mel spectrum feature vectors are concatenated to obtain a dictionary training sample set of dimension (N, L). N can be used to characterize the dimension of the mel spectrum feature vectors. L can be used to characterize the length of the dictionary training sample set.

[0159] According to embodiments of this disclosure, after obtaining the dictionary training sample set, a column sample selection algorithm can be used to select k columns from the dictionary training sample set as the number of atoms in the initial dictionary. The dimension of the initial dictionary can be (N, K). K can be much larger than N. In this case, the initial dictionary is an overcomplete dictionary.

[0160] According to embodiments of this disclosure, after obtaining an initial dictionary, a target dictionary can be obtained by processing the initial dictionary based on a dictionary learning algorithm. The dictionary learning algorithm can include at least one of the following: a dictionary construction algorithm based on a data model and a dictionary construction algorithm based on a learning algorithm. The dictionary construction algorithm based on a data model can include at least one of the following: wavelet transform and discrete cosine transform. The dictionary construction algorithm based on a learning algorithm can include the K-Singular Value Decomposition (K-SVD) algorithm. The K-SVD algorithm can perform singular value decomposition on the error term according to the principle of minimizing error, select the decomposition term that minimizes the error as the updated dictionary atom and corresponding atom coefficients, and obtain an optimized target dictionary that can sparsely represent the signal through continuous iteration. The dimension of the target dictionary can be (N, K).

[0161] According to embodiments of this disclosure, operation S422 may include the following operations.

[0162] Based on the tracking algorithm, the Mel-spectral feature vector is sparsely represented to obtain sparse coding coefficients. Then, based on the target dictionary and the sparse coding coefficients, the sparse spectral feature vector corresponding to the positional encoding is determined.

[0163] According to embodiments of this disclosure, sparse coding coefficients can be obtained by sparsely representing the Mel-spectral feature vector based on a tracking algorithm. The tracking algorithm may include at least one of the following: Matching Pursuit (OMP) algorithm and Orthogonal Matching Pursuit (OMP) algorithm. After obtaining the sparse coding coefficients, the sparse spectral feature vector corresponding to the positional coding can be determined based on the target dictionary and the sparse coding coefficients.

[0164] According to embodiments of this disclosure, since the sparse spectral feature vector is obtained by processing the Mel spectral feature vector corresponding to the position encoding based on the target dictionary, it is possible to achieve noise reduction and concise representation of the speech signal, thereby improving the recognition rate of speech recognition in noisy speech scenarios.

[0165] Figure 5 The illustration shows an example of a process of processing a Mel spectrum feature vector sequence to obtain a sparse spectrum feature vector sequence according to an embodiment of the present disclosure.

[0166] like Figure 5 As shown, in step 500, a dictionary training sample set 502 can be constructed based on at least one Mel-spectral feature vector 501. An initial dictionary 503 is determined based on the dictionary training sample set 502. The initial dictionary 503 is processed using a dictionary learning algorithm to obtain a target dictionary 504. A sparse representation of the Mel-spectral feature vector 501 is performed using a tracking algorithm to obtain sparse coding coefficients 505.

[0167] After obtaining the target dictionary 504 and sparse coding coefficients 505, the sparse spectral feature vectors 506 corresponding to the positional codes can be determined based on the target dictionary 504 and sparse coding coefficients 505. Based on the sparse spectral feature vectors 506 corresponding to each positional code, the sparse spectral feature vector sequence 507 is determined.

[0168] According to embodiments of this disclosure, operation S230 may include the following operations.

[0169] Determine the target Mel-spectral feature vector and target sparse-spectral feature vector corresponding to the positional encoding. Based on the target Mel-spectral feature vector, determine the first query matrix, first key matrix, and first value matrix. Based on the target sparse-spectral feature vector, determine the second query matrix, second key matrix, and second value matrix. Based on the first query matrix, first key matrix, first value matrix, second query matrix, second key matrix, and second value matrix, determine the fused feature vector corresponding to the positional encoding.

[0170] According to embodiments of this disclosure, a first query matrix, a first key matrix, and a first value matrix can be determined based on a target Mel-spectrum feature vector characterizing the original features of the signal. For example, the first query matrix can be determined based on the target Mel-spectrum feature vector and the query parameter matrix, the first key matrix can be determined based on the target Mel-spectrum feature vector and the key parameter matrix, and the first value matrix can be determined based on the target Mel-spectrum feature vector and the value parameter matrix.

[0171] According to embodiments of this disclosure, a second query matrix, a second key matrix, and a second value matrix can be determined based on the target sparse spectral feature vector representing the enhanced features. For example, the second query matrix can be determined based on the target sparse spectral feature vector and the query parameter matrix, the second key matrix can be determined based on the target sparse spectral feature vector and the key parameter matrix, and the second value matrix can be determined based on the target sparse spectral feature vector and the value parameter matrix.

[0172] According to embodiments of this disclosure, the target Mel-spectral feature vector and the target sparse spectral feature vector can be fused based on the first query matrix, the first key matrix, the first value matrix, the second query matrix, the second key matrix, and the second value matrix to obtain a fused feature vector corresponding to the position encoding.

[0173] According to embodiments of this disclosure, determining the fused feature vector corresponding to the position code based on the first query matrix, the first key matrix, the first value matrix, the second query matrix, the second key matrix, and the second value matrix may include the following operations.

[0174] Based on the first query matrix and the second key matrix, determine the first attention weight matrix. Based on the second query matrix and the first key matrix, determine the second attention weight matrix. Based on the first attention weight matrix and the second value matrix, obtain the first eigenvector. Based on the second attention weight matrix and the first value matrix, obtain the second eigenvector. Based on the first and second eigenvectors, obtain the fused eigenvector.

[0175] According to embodiments of this disclosure, after obtaining the first query matrix and the second key matrix, a first attention weight matrix representing the correlation between the target Mel-spectral feature vector and the target sparse spectral feature vector can be determined based on the first query matrix and the second key matrix. After obtaining the second query matrix and the first key matrix, a second attention weight matrix representing the correlation between the target Mel-spectral feature vector and the target sparse spectral feature vector can be determined based on the second query matrix and the first key matrix.

[0176] According to embodiments of this disclosure, a first eigenvector is determined after obtaining a first attention weight matrix and a second value matrix. A second eigenvector is determined after obtaining a second attention weight matrix and a first value matrix. After obtaining the first and second eigenvectors, a fused eigenvector can be obtained based on the first and second eigenvectors.

[0177] Figure 6 The illustration shows an example of a process of fusing a Mel-spectral feature vector sequence and a sparse-spectral feature vector sequence according to at least one positional encoding, according to an embodiment of the present disclosure, to obtain a fused feature vector sequence.

[0178] like Figure 6 As shown, in step 600, for each positional code in at least one positional code, a target Mel-spectral feature vector 601 and a target sparse spectral feature vector 602 corresponding to the positional code can be determined. Based on the target Mel-spectral feature vector 601, a first query matrix 603, a first key matrix 604, and a first value matrix 605 can be determined. Based on the target sparse spectral feature vector 602, a second query matrix 606, a second key matrix 607, and a second value matrix 608 can be determined.

[0179] After obtaining the first query matrix 603, the first key matrix 604, the second query matrix 606, and the second key matrix 607, the first attention weight matrix 609 can be determined based on the first query matrix 603 and the second key matrix 607. The second attention weight matrix 610 is then determined based on the second query matrix 606 and the first key matrix 604.

[0180] After obtaining the first attention weight matrix 609 and the second attention weight matrix 610, the first eigenvector 611 can be obtained based on the first attention weight matrix 609 and the second value matrix 608. The second eigenvector 612 can be obtained based on the second attention weight matrix 610 and the first value matrix 605. The fused eigenvector 613 can be obtained based on the first eigenvector 611 and the second eigenvector 612.

[0181] According to embodiments of this disclosure, the speech recognition method 200 can also be implemented based on a deep learning model, which includes a feature extraction layer, an attention layer, and a temporal classification layer. The speech recognition method 200 may include the following operations.

[0182] The speech data to be recognized is input into the feature extraction layer to obtain the Mel-spectral feature vector sequence and the sparse-spectral feature vector sequence. These two sequences are then input into the attention layer to obtain the fused feature vector sequence. Finally, the fused feature vector sequence is input into the temporal classification layer to obtain the speech recognition result.

[0183] According to embodiments of this disclosure, a feature extraction layer can be used to extract features from the speech data to be recognized, obtaining a Mel-spectrum feature vector sequence. Alternatively, the feature extraction layer can also be used to process the Mel-spectrum feature vector sequence to obtain a sparse-spectrum feature vector sequence. An attention layer can be used to fuse the Mel-spectrum feature vector sequence and the sparse-spectrum feature vector sequence to obtain a fused feature vector sequence. A temporal classification layer can include a Connectionist Temporal Classification (CTC) module. The temporal classification layer can be used to process the fused feature vector sequence to obtain the speech recognition result.

[0184] The above are merely exemplary embodiments, but are not limited thereto. Other speech recognition methods known in the art may also be included, as long as they can improve the accuracy of speech recognition.

[0185] Figure 7 A flowchart illustrating a method for training a speech recognition model according to an embodiment of the present disclosure is shown.

[0186] like Figure 7 As shown, the method 700 includes operations S710 to S740.

[0187] In operation S710, feature extraction is performed on at least one sample speech data to be identified, and sample Mel spectrum feature vector sequences corresponding to each of the at least one sample speech data to be identified are obtained. The sample Mel spectrum feature vector sequences include at least one sample position code and sample Mel spectrum feature vectors corresponding to each of the at least one sample position code. Each of the at least one sample speech data to be identified has a corresponding real speech recognition result.

[0188] In operation S720, the sample Mel spectrum feature vector sequence corresponding to each of the at least one sample speech data to be identified is processed to obtain the sample sparse spectrum feature vector sequence corresponding to each of the at least one sample speech data to be identified, wherein the sample sparse spectrum feature vector sequence includes at least one sample position code and a sample sparse spectrum feature vector corresponding to each of the at least one sample position code.

[0189] In operation S730, based on at least one sample position encoding, the sample Mel spectrum feature vector sequence and sample sparse spectrum feature vector sequence corresponding to each of the at least one sample speech data to be identified are respectively fused to obtain the sample fused feature vector sequence corresponding to each of the at least one sample speech data to be identified.

[0190] In operation S740, a deep learning model is trained using the real speech recognition results corresponding to at least one speech data sample to be recognized and the sample fusion feature vector sequence, to obtain a speech recognition model.

[0191] According to embodiments of this disclosure, the real speech recognition result can be used to characterize the real speech content corresponding to the speech data to be recognized. For descriptions of the speech data to be recognized, the sample Mel-spectral feature vector sequence, the sample position encoding, the sample Mel-spectral feature vector, the sample sparse spectral feature vector sequence, the sample sparse spectral feature vector, and the sample fused feature vector sequence, please refer to the above descriptions of the speech data to be recognized, the Mel-spectral feature vector sequence, the position encoding, the Mel-spectral feature vector, the sparse spectral feature vector sequence, the sparse spectral feature vector, and the fused feature vector sequence; these will not be repeated here.

[0192] According to embodiments of this disclosure, since the sample Mel-spectral feature vector sequence is obtained by feature extraction from the speech data to be recognized, the sample Mel-spectral feature vector sequence can be used to characterize the sample Mel-spectral features of the speech data to be recognized. Since the sample sparse spectral feature vector sequence is obtained by processing the sample Mel-spectral feature vector sequence, the sample sparse spectral feature vector sequence can be used to characterize the sample sparse spectral features of the speech data to be recognized. Based on this, by fusing the sample Mel-spectral feature vector sequence and the sample sparse spectral feature vector sequence, the sample Mel-spectral features and the sample sparse spectral features can complement each other, thus at least partially overcoming the technical problem in related technologies where the influence of background noise in noisy speech scenes easily leads to low speech recognition rates, improving the robustness of the speech recognition model, and thereby improving the accuracy of speech recognition.

[0193] The following is for reference. Figure 8 and Figure 9 The training method 700 for the speech recognition model according to an embodiment of the present invention will be further described.

[0194] According to embodiments of this disclosure, operation S730 may include the following operations.

[0195] The sample Mel spectrum feature vector sequence and sample sparse spectrum feature vector sequence corresponding to each of the at least one sample speech data to be identified are input into the attention layer to obtain the sample fusion feature vector sequence corresponding to each of the at least one sample speech data to be identified.

[0196] According to embodiments of this disclosure, a deep learning model may include an attention layer.

[0197] According to embodiments of this disclosure, the attention strategy can be used to focus on important information with high weight and ignore unimportant information with low weight, and can exchange information with other information by sharing important information, thereby realizing the transmission of important information.

[0198] According to embodiments of this disclosure, an attention layer can be determined based on an attention strategy. Sample Mel-spectral feature vector sequences and sample sparse-spectral feature vector sequences, each corresponding to at least one speech data sample to be recognized, are input into the attention layer. This allows the attention layer to perform feature fusion on the sample Mel-spectral feature vector sequences and sample sparse-spectral feature vector sequences, thereby enhancing the features and complementing each other's information, which is more conducive to model training.

[0199] According to embodiments of this disclosure, since the sample fusion feature vector sequence is obtained by inputting the sample Mel spectrum feature vector sequence and the sample sparse spectrum feature vector sequence corresponding to the speech data to be identified into the attention layer, the attention mechanism can focus on key information and complement each other, enabling the model to automatically learn clean and complete speech features.

[0200] Figure 8 The illustration shows an example diagram of the training process of a speech recognition model according to an embodiment of the present disclosure.

[0201] like Figure 8 As shown, in 800, the deep learning model 804 may include an attention layer 8041.

[0202] Feature extraction can be performed on at least one sample speech data 801 to be identified, to obtain sample Mel spectrum feature vector sequences 802 corresponding to each of the at least one sample speech data 801 to be identified.

[0203] The sample Mel spectrum feature vector sequence 802 corresponding to each of the at least one sample speech data 801 to be identified can be processed to obtain the sample sparse spectrum feature vector sequence 803 corresponding to each of the at least one sample speech data 801 to be identified.

[0204] The sample Mel spectrum feature vector sequence 802 and the sample sparse spectrum feature vector sequence 803 corresponding to each of the at least one sample speech data 801 to be identified are input into the attention layer 8041 to obtain the sample fusion feature vector sequence 805 corresponding to each of the at least one sample speech data 801 to be identified.

[0205] Using the real speech recognition results 806 corresponding to each of the speech data 801 to be recognized and the sample fusion feature vector sequence 805, a deep learning model 804 is trained to obtain a speech recognition model.

[0206] According to embodiments of this disclosure, operation S740 may include the following operations.

[0207] The sample fusion feature vector sequence corresponding to each of the at least one speech sample to be identified is input into the deep learning model to obtain the sample speech recognition result corresponding to each of the at least one speech sample to be identified. Based on the real speech recognition result and the sample speech recognition result corresponding to each of the at least one speech sample to be identified, the loss function value corresponding to each of the at least one speech sample to be identified is obtained. The model parameters of the deep learning model are adjusted based on the loss function value corresponding to each of the at least one speech sample to be identified until a predetermined termination condition is met, thus obtaining the speech recognition model.

[0208] According to embodiments of this disclosure, a loss function value can be obtained based on the loss function, using real speech recognition results and sample speech recognition results. The model parameters of the deep learning model are adjusted according to the loss function value until predetermined conditions are met. The deep learning model obtained under the predetermined conditions is determined as the speech recognition model. The specific form of the loss function can be set by those skilled in the art according to actual needs, and this disclosure does not limit this.

[0209] According to embodiments of this disclosure, inputting a sequence of sample fusion feature vectors corresponding to at least one sample speech data to be identified into a deep learning model to obtain sample speech recognition results corresponding to at least one sample speech data to be identified may include the following operations.

[0210] For each sample fusion feature vector sequence corresponding to at least one speech data sample to be recognized, the sample fusion feature vector sequence is input into a first one-dimensional convolutional layer to obtain a first intermediate sample feature vector sequence. The first intermediate sample feature vector sequence is input into a second one-dimensional convolutional layer to obtain a second intermediate sample feature vector sequence. The second intermediate sample feature vector sequence is input into a max pooling layer to obtain a third intermediate sample feature vector sequence. The third intermediate sample feature vector sequence is input into a fully connected layer to obtain a fourth intermediate sample feature vector sequence. The fourth intermediate sample feature vector sequence is input into a normalization layer to obtain the sample speech recognition result.

[0211] According to embodiments of this disclosure, the deep learning model may further include a temporal classification layer. The temporal classification layer may include a first one-dimensional convolutional layer, a second one-dimensional convolutional layer, a max-pooling layer, a fully connected layer, and a normalization layer.

[0212] According to embodiments of this disclosure, a deep learning model may include a Convolutional Neural Network (CNN) module and a Connectionist Temporal Classification (CTC) module. The CNN module may include at least a first convolutional block. The first convolutional block may include a first one-dimensional convolutional layer, a second one-dimensional convolutional layer, and a first max-pooling layer. The CTC module may include a fully connected layer and a normalization layer. The one-dimensional convolutional layer can be understood as Conv1D.

[0213] According to embodiments of this disclosure, the convolutional neural network module may further include a second convolutional block and a third convolutional block. The second convolutional block may include a third one-dimensional convolutional layer, a fourth one-dimensional convolutional layer, and a second max-pooling layer. The third convolutional block may include a fifth one-dimensional convolutional layer, a sixth one-dimensional convolutional layer, and a third max-pooling layer.

[0214] According to embodiments of this disclosure, since the first intermediate sample feature vector sequence is obtained by inputting the sample fusion feature vector sequence into the first one-dimensional convolutional layer, and the second intermediate sample feature vector sequence is obtained by inputting the first intermediate sample feature vector sequence into the second one-dimensional convolutional layer, the one-dimensional convolutional layer is more suitable for speech signals, thereby improving the robustness of the speech recognition model and thus improving the accuracy of speech recognition.

[0215] Figure 9 The illustration shows an example diagram of the training process of a speech recognition model according to another embodiment of the present disclosure.

[0216] like Figure 9 As shown, in 900, the deep learning model 904 may include an attention layer 904_1, a convolutional neural network module 904_2, and a temporal classification module 904_3. The convolutional neural network module 904_2 may include a first one-dimensional convolutional layer 904_21, a second one-dimensional convolutional layer 904_22, and a max-pooling layer 904_23. The temporal classification module 904_3 may include a fully connected layer 904_31 and a normalization layer 904_32.

[0217] Feature extraction can be performed on at least one sample speech data 901 to be identified, to obtain sample Mel spectrum feature vector sequences 902 corresponding to each of the at least one sample speech data 901 to be identified.

[0218] The sample Mel spectrum feature vector sequence 902 corresponding to each of the at least one sample speech data 901 to be identified is processed to obtain the sample sparse spectrum feature vector sequence 903 corresponding to each of the at least one sample speech data 901 to be identified.

[0219] The sample Mel spectrum feature vector sequence 902 and the sample sparse spectrum feature vector sequence 903 corresponding to each of the at least one sample speech data 901 to be identified are input into the attention layer 904_1 to obtain the sample fusion feature vector sequence 905 corresponding to each of the at least one sample speech data 901 to be identified.

[0220] The sample fusion feature vector sequence 905 is input into the first one-dimensional convolutional layer 904_21 to obtain the first intermediate sample feature vector sequence 906. The first intermediate sample feature vector sequence 906 is input into the second one-dimensional convolutional layer 904_22 to obtain the second intermediate sample feature vector sequence 907. The second intermediate sample feature vector sequence 907 is input into the max pooling layer 904_23 to obtain the third intermediate sample feature vector sequence 908.

[0221] The third intermediate sample feature vector sequence 908 is input into the fully connected layer 904_31 to obtain the fourth intermediate sample feature vector sequence 909. The fourth intermediate sample feature vector sequence 909 is then input into the normalization layer 90432 to obtain the sample speech recognition result 910.

[0222] Based on the loss function 912, the loss function value 913 corresponding to each of the at least one sample speech data 901 to be identified is obtained according to the real speech recognition result 911 and the sample speech recognition result 910 corresponding to each of the at least one sample speech data 901 to be identified.

[0223] The model parameters of the deep learning model 904 are adjusted according to the loss function value 913 corresponding to each of the at least one speech data sample 901 to be recognized, until the predetermined termination condition is met, and a speech recognition model is obtained.

[0224] The above are merely exemplary embodiments, but are not limited thereto. Other training methods for speech recognition models known in the art may also be included, as long as they can improve the robustness of the speech recognition model and the accuracy of speech recognition.

[0225] Figure 10 A block diagram of a speech recognition device according to an embodiment of the present disclosure is shown schematically.

[0226] like Figure 10 As shown, the speech recognition device 1000 may include a first feature extraction module 1010, a first processing module 1020, a first fusion processing module 1030, and a second processing module 1040.

[0227] The first feature extraction module 1010 is used to extract features from the speech data to be recognized to obtain a Mel spectrum feature vector sequence, wherein the Mel spectrum feature vector sequence includes at least one position code and a Mel spectrum feature vector corresponding to each of the at least one position code.

[0228] The first processing module 1020 is used to process the Mel spectrum feature vector sequence to obtain a sparse spectrum feature vector sequence, wherein the sparse spectrum feature vector sequence includes at least one position code and a sparse spectrum feature vector corresponding to each of the at least one position code.

[0229] The first fusion processing module 1030 is used to fuse the Mel spectrum feature vector sequence and the sparse spectrum feature vector sequence according to at least one position code to obtain a fused feature vector sequence.

[0230] The second processing module 1040 is used to process the fused feature vector sequence to obtain the speech recognition result.

[0231] According to embodiments of this disclosure, for each of the at least one location codes, the first fusion processing module 1030 may include a first determining submodule, a second determining submodule, a third determining submodule, and a fourth determining submodule.

[0232] The first determining submodule is used to determine the target Mel-spectral feature vector and the target sparse-spectral feature vector corresponding to the position encoding.

[0233] The second determining submodule is used to determine the first query matrix, the first key matrix, and the first value matrix based on the target Mel spectrum feature vector.

[0234] The third determination submodule is used to determine the second query matrix, the second key matrix, and the second value matrix based on the target sparse spectral feature vector.

[0235] The fourth determination submodule is used to determine the fused feature vector corresponding to the position encoding based on the first query matrix, the first key matrix, the first value matrix, the second query matrix, the second key matrix, and the second value matrix.

[0236] According to embodiments of this disclosure, the fourth determining submodule may include a first determining unit, a second determining unit, a first obtaining unit, a second obtaining unit, and a third obtaining unit.

[0237] The first determining unit is used to determine the first attention weight matrix based on the first query matrix and the second key matrix.

[0238] The second determining unit is used to determine the second attention weight matrix based on the second query matrix and the first key matrix.

[0239] The first obtaining unit is used to obtain the first feature vector based on the first attention weight matrix and the second value matrix.

[0240] The second obtaining unit is used to obtain the second feature vector based on the second attention weight matrix and the first value matrix.

[0241] The third obtaining unit is used to obtain the fused feature vector based on the first feature vector and the second feature vector.

[0242] According to embodiments of this disclosure, the first processing module 1020 may include a fifth determining submodule, a first processing submodule, and a sixth determining submodule.

[0243] The fifth determination submodule is used to determine the target dictionary based on the Mel spectrum feature vector sequence.

[0244] The first processing submodule is used to process the Mel spectral feature vector corresponding to each position code in at least one position code, according to the target dictionary, to obtain the sparse spectral feature vector corresponding to the position code.

[0245] The sixth determination submodule is used to determine the sparse spectral feature vector sequence based on the sparse spectral feature vector corresponding to each position.

[0246] According to embodiments of this disclosure, the fifth determining submodule may include a construction unit, a third determining unit, and a first processing unit.

[0247] The building unit is used to construct a dictionary training sample set based on at least one Mel spectrum feature vector.

[0248] The third determining unit is used to determine the initial dictionary based on the dictionary training sample set.

[0249] The first processing unit is used to process the initial dictionary based on the dictionary learning algorithm to obtain the target dictionary.

[0250] According to embodiments of this disclosure, the first processing submodule may include a fourth obtaining unit and a fourth determining unit.

[0251] The fourth acquisition unit is used to perform sparse representation of the Mel spectrum feature vector based on the tracking algorithm to obtain sparse coding coefficients.

[0252] The fourth determining unit is used to determine the sparse spectral feature vector corresponding to the positional encoding based on the target dictionary and sparse coding coefficients.

[0253] According to embodiments of this disclosure, the first feature extraction module 1010 may include a second processing submodule, a seventh determination submodule, and a feature extraction submodule.

[0254] The second processing submodule is used to preprocess the speech data to be recognized to obtain Mel spectrum sequence data, wherein the Mel spectrum sequence data includes at least one Mel spectrum data.

[0255] The seventh determination submodule is used to determine intermediate coding sequence data based on the Mel spectrum sequence data and the position coding sequence data, wherein the position coding sequence data includes at least one position code.

[0256] The feature extraction submodule is used to extract features from the intermediate encoded sequence data to obtain the Mel spectrum feature vector sequence.

[0257] According to embodiments of this disclosure, the second processing submodule may include a second processing unit, a third processing unit, and a feature extraction unit.

[0258] The second processing unit is used to perform data augmentation processing on the speech data to be recognized, so as to obtain augmented speech data.

[0259] The third processing unit is used to perform windowing processing on the enhanced speech data to obtain the target speech data.

[0260] The feature extraction unit is used to extract features from the target speech data to obtain Mel spectrum sequence data.

[0261] According to embodiments of this disclosure, the speech recognition device can be implemented based on a deep learning model, which may include a feature extraction layer, an attention layer, and a temporal classification layer. The device may also include a first input module, a second input module, and a third input module.

[0262] The first input module is used to input the speech data to be recognized into the feature extraction layer to obtain the Mel spectrum feature vector sequence and the sparse spectrum feature vector sequence.

[0263] The second input module is used to input the Mel spectrum feature vector sequence and the sparse spectrum feature vector sequence into the attention layer to obtain the fused feature vector sequence.

[0264] The third input module is used to input the fused feature vector sequence into the temporal classification layer to obtain the speech recognition result. Figure 11 A block diagram of a training apparatus for a speech recognition model according to an embodiment of the present disclosure is shown schematically.

[0265] like Figure 11 As shown, the training device 1100 for the speech recognition model may include a second feature extraction module 1110, a third processing module 1120, a second fusion processing module 1130, and a training module 1140.

[0266] The second feature extraction module 1110 is used to extract features from at least one sample speech data to be identified, and obtain a sample Mel spectrum feature vector sequence corresponding to each of the at least one sample speech data to be identified. The sample Mel spectrum feature vector sequence includes at least one sample position code and a sample Mel spectrum feature vector corresponding to each of the at least one sample position code. Each of the at least one sample speech data to be identified has a corresponding real speech recognition result.

[0267] The third processing module 1120 is used to process the sample Mel spectrum feature vector sequence corresponding to each of the at least one sample speech data to be identified, to obtain the sample sparse spectrum feature vector sequence corresponding to each of the at least one sample speech data to be identified, wherein the sample sparse spectrum feature vector sequence includes at least one sample position code and a sample sparse spectrum feature vector corresponding to each of the at least one sample position code.

[0268] The second fusion processing module 1130 is used to perform fusion processing on the sample Mel spectrum feature vector sequence and the sample sparse spectrum feature vector sequence corresponding to each of the at least one sample speech data to be identified, according to at least one sample position encoding, to obtain the sample fusion feature vector sequence corresponding to each of the at least one sample speech data to be identified.

[0269] The training module 1140 is used to train a deep learning model using the real speech recognition results and sample fusion feature vector sequences corresponding to at least one speech data sample to be recognized, and to obtain a speech recognition model.

[0270] According to embodiments of this disclosure, the deep learning model includes an attention layer.

[0271] According to embodiments of this disclosure, the second fusion processing module 1130 may include a first input submodule.

[0272] The first input submodule is used to input the sample Mel spectrum feature vector sequence and the sample sparse spectrum feature vector sequence corresponding to each of the at least one sample speech data to be identified into the attention layer to obtain the sample fusion feature vector sequence corresponding to each of the at least one sample speech data to be identified.

[0273] According to embodiments of this disclosure, the training module 1140 may include a second input submodule, an acquisition submodule, and an adjustment submodule.

[0274] The second input submodule is used to input the sample fusion feature vector sequence corresponding to each of the at least one sample speech data to be identified into the deep learning model to obtain the sample speech recognition result corresponding to each of the at least one sample speech data to be identified.

[0275] The submodule is used to obtain the loss function value corresponding to each of the at least one speech data to be identified, based on the real speech recognition result and the sample speech recognition result corresponding to each of the at least one speech data to be identified.

[0276] The adjustment submodule is used to adjust the model parameters of the deep learning model according to the loss function values ​​corresponding to at least one speech data sample to be recognized, until a predetermined termination condition is met, thus obtaining a speech recognition model.

[0277] According to embodiments of this disclosure, the deep learning model further includes a temporal classification layer. The temporal classification layer may include a first one-dimensional convolutional layer, a second one-dimensional convolutional layer, a max-pooling layer, a fully connected layer, and a normalization layer.

[0278] According to embodiments of this disclosure, for each sample fusion feature vector sequence in the sample fusion feature vector sequence corresponding to at least one sample speech data to be identified, the second input submodule may include a first input unit, a second input unit, a third input unit, a fourth input unit, and a fifth input unit.

[0279] The first input unit is used to input the sample fusion feature vector sequence into the first one-dimensional convolutional layer to obtain the first intermediate sample feature vector sequence.

[0280] The second input unit is used to input the first intermediate sample feature vector sequence into the second one-dimensional convolutional layer to obtain the second intermediate sample feature vector sequence.

[0281] The third input unit is used to input the second intermediate sample feature vector sequence into the max pooling layer to obtain the third intermediate sample feature vector sequence.

[0282] The fourth input unit is used to input the third intermediate sample feature vector sequence into the fully connected layer to obtain the fourth intermediate sample feature vector sequence.

[0283] The fifth input unit is used to input the feature vector sequence of the fourth intermediate sample into the normalization layer to obtain the sample speech recognition result.

[0284] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuitry, such as Field Programmable Gate Arrays (FPGAs), Programmable Logic Arrays (PLAs), Systems-on-Chip, Systems-on-Substrate, Systems-on-Package, Application-Specific Integrated Circuits (ASICs), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0285] For example, any plurality of the first feature extraction module 1010, the first processing module 1020, the first fusion processing module 1030, and the second processing module 1040 can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits can be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this disclosure, at least one of the first feature extraction module 1010, the first processing module 1020, the first fusion processing module 1030, and the second processing module 1040 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the first feature extraction module 1010, the first processing module 1020, the first fusion processing module 1030, and the second processing module 1040 may be at least partially implemented as a computer program module, which can perform corresponding functions when the computer program module is run.

[0286] It should be noted that the speech recognition device part in the embodiments of this disclosure corresponds to the speech recognition method part in the embodiments of this disclosure. For a detailed description of the speech recognition device part, please refer to the speech recognition method part, which will not be repeated here.

[0287] For example, any plurality of the second feature extraction module 1110, the third processing module 1120, the second fusion processing module 1130, and the training module 1140 can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits can be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of the present disclosure, at least one of the second feature extraction module 1110, the third processing module 1120, the second fusion processing module 1130, and the training module 1140 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the second feature extraction module 1110, the third processing module 1120, the second fusion processing module 1130, and the training module 1140 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0288] It should be noted that the training device part of the speech recognition model in the embodiments of this disclosure corresponds to the training method part of the speech recognition model in the embodiments of this disclosure. For a detailed description of the training device part of the speech recognition model, please refer to the training method part of the speech recognition model, which will not be repeated here.

[0289] Figure 12 A block diagram of an electronic device suitable for implementing a speech recognition method and a training method for a speech recognition model according to embodiments of the present disclosure is shown schematically. Figure 12 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0290] like Figure 12As shown, a computer electronic device 1200 according to an embodiment of the present disclosure includes a processor 1201, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1202 or a program loaded from a storage portion 1209 into a random access memory (RAM) 1203. The processor 1201 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 1201 may also include onboard memory for caching purposes. The processor 1201 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0291] RAM 1203 stores various programs and data required for the operation of electronic device 1200. Processor 1201, ROM 1202, and RAM 1203 are interconnected via bus 1204. Processor 1201 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 1202 and / or RAM 1203. It should be noted that the programs may also be stored in one or more memories other than ROM 1202 and RAM 1203. Processor 1201 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0292] According to embodiments of this disclosure, the electronic device 1200 may further include an input / output (I / O) interface 1205, which is also connected to the bus 1204. The electronic device 1200 may also include one or more of the following components connected to the I / O interface 1205: an input section 1206 including a keyboard, mouse, etc.; an output section 1207 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1208 including a hard disk, etc.; and a communication section 1209 including a network interface card such as a LAN card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet. A drive 1210 is also connected to the I / O interface 1205 as needed. A removable medium 1211, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 1210 as needed so that computer programs read from it can be installed into the storage section 1208 as needed.

[0293] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1209, and / or installed from removable medium 1211. When the computer program is executed by processor 1201, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0294] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0295] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0296] For example, according to embodiments of this disclosure, a computer-readable storage medium may include the ROM 1202 and / or RAM 1203 described above and / or one or more memories other than ROM 1202 and RAM 1203.

[0297] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the speech recognition method and the speech recognition model training method provided in the embodiments of this disclosure.

[0298] When the computer program is executed by the processor 1201, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0299] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 1209, and / or installed from the removable medium 1211. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0300] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0301] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features recited in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not expressly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0302] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A speech recognition method, comprising: Feature extraction is performed on the speech data to be recognized to obtain a Mel spectrum feature vector sequence, wherein the Mel spectrum feature vector sequence includes at least one position code and a Mel spectrum feature vector corresponding to each of the at least one position code; The Mel spectrum feature vector sequence is processed to obtain a sparse spectrum feature vector sequence, wherein the sparse spectrum feature vector sequence includes at least one position code and a sparse spectrum feature vector corresponding to each of the at least one position code; Using the attention layer in a deep learning model, and based on at least one positional encoding, the Mel-spectral feature vector sequence and the sparse-spectral feature vector sequence are fused to focus on the key information of the Mel-spectral feature vectors and the sparse-spectral feature vectors. Clean and complete speech features are learned through mutual complementarity, resulting in a fused feature vector sequence. The fused feature vector sequence is processed to obtain the speech recognition result.

2. The method according to claim 1, wherein, The step of fusing the Mel-spectral feature vector sequence and the sparse spectral feature vector sequence according to the at least one positional encoding to obtain a fused feature vector sequence includes: For each of the at least one position code, Determine the target Mel-spectral feature vector and the target sparse-spectral feature vector corresponding to the position encoding; Based on the target Mel spectrum feature vector, determine the first query matrix, the first key matrix, and the first value matrix; Based on the target sparse spectral feature vector, determine the second query matrix, the second key matrix, and the second value matrix; and Based on the first query matrix, the first key matrix, the first value matrix, the second query matrix, the second key matrix, and the second value matrix, determine the fusion feature vector corresponding to the position code.

3. The method according to claim 2, wherein, The step of determining the fused feature vector corresponding to the position code based on the first query matrix, the first key matrix, the first value matrix, the second query matrix, the second key matrix, and the second value matrix includes: Based on the first query matrix and the second key matrix, determine the first attention weight matrix; The second attention weight matrix is ​​determined based on the second query matrix and the first key matrix; The first feature vector is obtained based on the first attention weight matrix and the second value matrix; Based on the second attention weight matrix and the first value matrix, the second feature vector is obtained; and The fused feature vector is obtained based on the first feature vector and the second feature vector.

4. The method according to any one of claims 1 to 3, wherein, The process of processing the Mel spectrum feature vector sequence to obtain the sparse spectrum feature vector sequence includes: The target dictionary is determined based on the Mel spectrum feature vector sequence; For each of the at least one position code, Based on the target dictionary, the Mel-spectral feature vector corresponding to the position code is processed to obtain the sparse spectral feature vector corresponding to the position code; and The sparse spectral feature vector sequence is determined based on the sparse spectral feature vector corresponding to each of the position codes.

5. The method according to claim 4, wherein, The step of determining the target dictionary based on the Mel spectrum feature vector sequence includes: Construct a dictionary training sample set based on at least one of the Mel spectrum feature vectors; Based on the dictionary training sample set, determine the initial dictionary; and The initial dictionary is processed using a dictionary learning algorithm to obtain the target dictionary.

6. The method according to claim 4, wherein, The step of processing the Mel-spectral feature vector corresponding to the position code according to the target dictionary to obtain the sparse spectral feature vector corresponding to the position code includes: Based on the tracking algorithm, sparse representation of the Mel-spectral feature vector is performed to obtain sparse coding coefficients; and Based on the target dictionary and the sparse coding coefficients, the sparse spectral feature vector corresponding to the positional coding is determined.

7. The method according to any one of claims 1 to 3, wherein, The process of extracting features from the speech data to be recognized to obtain the Mel spectrum feature vector sequence includes: The speech data to be recognized is preprocessed to obtain Mel spectrum sequence data, wherein the Mel spectrum sequence data includes at least one Mel spectrum data. Based on the Mel spectrum sequence data and the position-coded sequence data, intermediate coded sequence data is determined, wherein the position-coded sequence data includes the at least one position code; and Feature extraction is performed on the intermediate encoded sequence data to obtain the Mel spectrum feature vector sequence.

8. The method according to claim 7, wherein, The preprocessing of the speech data to be recognized to obtain Mel spectrum sequence data includes: The speech data to be recognized is subjected to data augmentation processing to obtain augmented speech data; The enhanced speech data is windowed to obtain the target speech data; and Feature extraction is performed on the target speech data to obtain the Mel spectrum sequence data.

9. The method according to any one of claims 1 to 3, wherein, The speech recognition method is implemented based on the deep learning model, which includes a feature extraction layer, the attention layer, and a temporal classification layer. The method further includes: The speech data to be recognized is input into the feature extraction layer to obtain the Mel spectrum feature vector sequence and the sparse spectrum feature vector sequence; The Mel-spectral feature vector sequence and the sparse spectral feature vector sequence are input into the attention layer to obtain the fused feature vector sequence; The fused feature vector sequence is input into the temporal classification layer to obtain the speech recognition result.

10. A method for training a speech recognition model, comprising: Feature extraction is performed on at least one speech data sample to be identified to obtain a sample Mel spectrum feature vector sequence corresponding to each of the at least one speech data sample to be identified. The sample Mel spectrum feature vector sequence includes at least one sample position code and a sample Mel spectrum feature vector corresponding to each of the at least one sample position code. Each of the at least one speech data sample to be identified corresponds to a real speech recognition result. The sample Mel-spectral feature vector sequence corresponding to each of the at least one sample speech data to be identified is processed to obtain the sample sparse spectral feature vector sequence corresponding to each of the at least one sample speech data to be identified, wherein the sample sparse spectral feature vector sequence includes the at least one sample position code and the sample sparse spectral feature vector corresponding to each of the at least one sample position code. Using the attention layer in a deep learning model, based on the at least one sample position encoding, the sample Mel-spectral feature vector sequence and the sample sparse-spectral feature vector sequence corresponding to each of the at least one speech data to be identified are fused respectively. This focuses the key information of the sample Mel-spectral feature vector and the sample sparse-spectral feature vector, and learns clean and complete speech features through mutual complementarity, resulting in a sample fused feature vector sequence corresponding to each of the at least one speech data to be identified; and The deep learning model is trained using the real speech recognition results corresponding to each of the at least one speech data to be recognized sample and the sample fusion feature vector sequence to obtain a speech recognition model.

11. The method according to claim 10, wherein, The deep learning model includes the attention layer; The step of fusing the sample Mel-spectral feature vector sequence and the sample sparse spectral feature vector sequence corresponding to each of the at least one sample speech data to be identified, based on the at least one sample position encoding, to obtain the sample fused feature vector sequence corresponding to each of the at least one sample speech data to be identified, includes: The sample Mel-spectral feature vector sequence and the sample sparse-spectral feature vector sequence corresponding to each of the at least one speech data to be identified are input into the attention layer to obtain the sample fusion feature vector sequence corresponding to each of the at least one speech data to be identified.

12. The method according to claim 10 or 11, wherein, The step of training a deep learning model using the real speech recognition results corresponding to each of the at least one speech data sample to be recognized and the sample fusion feature vector sequence to obtain a speech recognition model includes: The sample fusion feature vector sequence corresponding to each of the at least one unidentified sample speech data is input into the deep learning model to obtain the sample speech recognition result corresponding to each of the at least one unidentified sample speech data. Based on the real speech recognition result and the sample speech recognition result corresponding to each of the at least one sample speech data to be recognized, the loss function value corresponding to each of the at least one sample speech data to be recognized is obtained; and The model parameters of the deep learning model are adjusted according to the loss function values ​​corresponding to each of the at least one speech data sample to be recognized, until a predetermined termination condition is met, thereby obtaining the speech recognition model.

13. The method according to claim 12, wherein, The deep learning model also includes a temporal classification layer, which includes a first one-dimensional convolutional layer, a second one-dimensional convolutional layer, a max pooling layer, a fully connected layer, and a normalization layer. The step of inputting the sample fusion feature vector sequence corresponding to each of the at least one unidentified sample speech data into the deep learning model to obtain the sample speech recognition result corresponding to each of the at least one unidentified sample speech data includes: For each sample fusion feature vector sequence in the sample fusion feature vector sequence corresponding to the at least one speech data sample to be identified, The sample fusion feature vector sequence is input into the first one-dimensional convolutional layer to obtain the first intermediate sample feature vector sequence. The first intermediate sample feature vector sequence is input into the second one-dimensional convolutional layer to obtain the second intermediate sample feature vector sequence. The second intermediate sample feature vector sequence is input into the max pooling layer to obtain the third intermediate sample feature vector sequence; The third intermediate sample feature vector sequence is input into the fully connected layer to obtain the fourth intermediate sample feature vector sequence; and The fourth intermediate sample feature vector sequence is input into the normalization layer to obtain the sample speech recognition result.

14. A voice recognition device, comprising: The first feature extraction module is used to extract features from the speech data to be recognized to obtain a Mel spectrum feature vector sequence, wherein the Mel spectrum feature vector sequence includes at least one position code and a Mel spectrum feature vector corresponding to each of the at least one position code; The first processing module is used to process the Mel spectrum feature vector sequence to obtain a sparse spectrum feature vector sequence, wherein the sparse spectrum feature vector sequence includes at least one position code and a sparse spectrum feature vector corresponding to each of the at least one position code. The first fusion processing module is used to fuse the Mel-spectral feature vector sequence and the sparse spectral feature vector sequence using the attention layer in the deep learning model, based on the at least one positional encoding. This is to focus on the key information of the Mel-spectral feature vector and the sparse spectral feature vector, and to learn clean and complete speech features through mutual complementarity, thereby obtaining a fused feature vector sequence. The second processing module is used to process the fused feature vector sequence to obtain the speech recognition result.

15. A training device for a speech recognition model, comprising: The second feature extraction module is used to extract features from at least one sample speech data to be identified, and obtain a sample Mel spectrum feature vector sequence corresponding to each of the at least one sample speech data to be identified. The sample Mel spectrum feature vector sequence includes at least one sample position code and a sample Mel spectrum feature vector corresponding to each of the at least one sample position code. Each of the at least one sample speech data to be identified has a real speech recognition result. The third processing module is used to process the sample Mel spectrum feature vector sequence corresponding to each of the at least one sample speech data to be identified, to obtain the sample sparse spectrum feature vector sequence corresponding to each of the at least one sample speech data to be identified, wherein the sample sparse spectrum feature vector sequence includes the at least one sample position code and the sample sparse spectrum feature vector corresponding to each of the at least one sample position code. The second fusion processing module is used to utilize the attention layer in the deep learning model to fuse the sample spectral feature vector sequence and the sample sparse spectral feature vector sequence corresponding to each of the at least one sample speech data to be identified, based on the at least one sample position encoding. This is to focus on the key information of the sample spectral feature vector and the sample sparse spectral feature vector, and to learn clean and complete speech features through mutual complementarity, thereby obtaining a sample fusion feature vector sequence corresponding to each of the at least one sample speech data to be identified; and The training module is used to train a deep learning model using the real speech recognition results corresponding to each of the at least one speech data to be recognized sample and the sample fusion feature vector sequence, so as to obtain a speech recognition model.

16. An electronic device comprising: One or more processors; Memory, used to store one or more instructions. When the one or more instructions are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method of any one of claims 1 to 9 or any one of claims 10 to 13.

17. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 9 or any one of claims 10 to 13.

18. A computer program product comprising computer-executable instructions, which, when executed, are used to implement the method of any one of claims 1 to 9 or any one of claims 10 to 13.

Images