Speech recognition method and apparatus, electronic device, and storage medium

Abstract

The present disclosure discloses a speech recognition method and device, electronic equipment and storage medium, and particularly relates to the technical field of artificial intelligence. The specific implementation scheme is as follows: a spectrum graph corresponding to a speech to be recognized is obtained; global features corresponding to each feature point in the spectrum graph are determined according to feature values of each feature point in the spectrum graph; frequency domain features of each frequency band in the spectrum graph are determined according to the feature values of each feature point in the spectrum graph and the frequency band to which each feature point belongs; the spectrum features to be recognized are determined according to the global features corresponding to the spectrum graph and the frequency domain features of each frequency band; and the spectrum features are recognized to determine a recognition result corresponding to the speech. Thus, in the process of speech recognition, not only the relationship between each feature point in the spectrum graph is considered, but also the frequency domain features of each frequency band in the spectrum graph are considered, thereby improving the accuracy and integrity of the speech recognition result.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computers, in particular to the technical field of artificial intelligence such as deep learning and speech technology, and especially to a speech recognition method and device, an electronic device and a storage medium. BACKGROUND

[0002] With the vigorous development of computer technology, the field of artificial intelligence has also developed rapidly, and the application of speech recognition technology is also becoming more and more widespread. In the related technology, a speech recognition model is usually used for speech recognition, and the accuracy of the model depends on the size of the training data and the number of pronunciation types covered by the speech data in the training data. For example, if the size of the training data is large enough, but the training data is all adult speech data, then when using the model trained by the training data to recognize child voices, the accuracy of the recognition is low. Therefore, how to improve the accuracy of speech recognition has become a problem to be solved. SUMMARY

[0003] The present disclosure provides a speech recognition method and device, an electronic device and a storage medium.

[0004] In one aspect of the present disclosure, a speech recognition method is provided, comprising:

[0005] obtaining a frequency spectrum corresponding to the speech to be recognized;

[0006] determining a global feature corresponding to each feature point in the frequency spectrum according to the feature value of each feature point in the frequency spectrum;

[0007] determining a frequency domain feature of each frequency band in the frequency spectrum according to the feature value of each feature point in the frequency spectrum and the frequency band to which each feature point belongs;

[0008] determining a spectrum feature to be recognized according to the global feature corresponding to the frequency spectrum and the frequency domain feature of each frequency band;

[0009] recognizing the spectrum feature to determine a recognition result corresponding to the speech.

[0010] In another aspect of the present disclosure, a speech recognition model is provided, comprising: N speech feature extraction modules and a speech recognition module connected in sequence, wherein N is a positive integer greater than 1.

[0011] Each of the speech feature extraction modules comprises a convolutional network, a global attention network and a frequency domain attention network;

[0012] The convolutional network is configured to perform convolutional processing on the obtained speech feature to obtain a new speech feature;

[0013] The global attention network is configured to process the new speech feature to output a global feature of the new speech feature.

[0014] The frequency domain attention network is configured to process the new speech feature to determine a frequency domain feature of each frequency band in the new speech feature.

[0015] The speech recognition module is configured to recognize the global feature and the frequency domain feature output by the Nth speech feature extraction module to output a recognition result.

[0016] Another aspect of the present disclosure provides a speech recognition device, comprising:

[0017] The acquisition module is configured to acquire a spectrum graph corresponding to a speech to be recognized.

[0018] The first determination module is configured to determine a global feature corresponding to each feature point in the spectrum graph according to a feature value of each feature point in the spectrum graph.

[0019] The second determination module is configured to determine a frequency domain feature of each frequency band in the spectrum graph according to the feature value of each feature point in the spectrum graph and a frequency band to which each feature point belongs.

[0020] The third determination module is configured to determine a spectrum feature to be recognized according to the global feature corresponding to the spectrum graph and the frequency domain feature of each frequency band.

[0021] The recognition module is configured to recognize the spectrum feature to determine a recognition result corresponding to the speech.

[0022] Another aspect of the present disclosure provides an electronic device, comprising:

[0023] at least one processor; and

[0024] a memory in communication with the at least one processor; wherein

[0025] The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the speech recognition method of the above-mentioned aspect embodiment.

[0026] Another aspect of the present disclosure provides a non-transitory computer readable storage medium storing computer instructions, and the computer instructions are used to make the computer perform the speech recognition method of the above-mentioned aspect embodiment.

[0027] In another aspect of the present disclosure, a computer program product is provided, which comprises a computer program that, when executed by a processor, implements the speech recognition method described in the above-mentioned aspect of the embodiment.

[0028] The speech recognition method, device, electronic device, and storage medium provided by the present disclosure can first acquire a spectrum graph corresponding to the speech to be recognized, then determine a correlation matrix between each feature point in the spectrum graph according to the feature value of each feature point in the spectrum graph, then determine the frequency domain feature of each frequency band in the spectrum graph according to the feature value of each feature point in the spectrum graph and the frequency band to which each feature point belongs, then determine the spectrum feature to be recognized according to the feature value of each feature point, the correlation matrix, and the frequency domain feature of each frequency band, and finally recognize the spectrum feature to determine the recognition result corresponding to the speech to be recognized. Thus, in the process of speech recognition, not only the relationship between each feature point in the spectrum graph is considered, but also the frequency domain feature of each frequency band in the spectrum graph is considered, thereby improving the accuracy and integrity of the speech recognition result.

[0029] It should be understood that the content described in this part is not intended to identify key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0030] The accompanying drawings are used to better understand the present scheme and do not limit the present disclosure. Among them:

[0031] Figure 1 A flowchart of a speech recognition method according to an embodiment of the present disclosure is provided;

[0032] Figure 2 A flowchart of a speech recognition method according to another embodiment of the present disclosure is provided;

[0033] Figure 3 A flowchart of a speech recognition method according to another embodiment of the present disclosure is provided;

[0034] Figure 4 A structure diagram of a speech recognition model according to an embodiment of the present disclosure is provided;

[0035] Figure 4A A structure diagram of a speech recognition model according to an embodiment of the present disclosure is provided;

[0036] Figure 5 A structure diagram of a speech recognition device according to another embodiment of the present disclosure is provided;

[0037] Figure 6 A block diagram of an electronic device for implementing the speech recognition method according to an embodiment of the present disclosure is provided. Detailed Implementation

[0038] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0039] Artificial intelligence (AI) is the study of enabling computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It involves both hardware and software technologies. AI hardware technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing. AI software technologies mainly include computer vision, speech recognition, natural language processing, as well as machine learning, deep learning, big data processing, and knowledge graph technologies.

[0040] Deep learning refers to multi-layered artificial neural networks and the methods for training them. A single layer of a neural network takes a large number of matrix numbers as input, applies weights through non-linear activation methods, and then generates another set of data as output. By using an appropriate number of matrices, multiple layers are linked together to form the "brain" of the neural network, performing precise and complex processing, much like how humans identify objects and label images.

[0041] The speech recognition method, apparatus, electronic device, and storage medium of this disclosure are described below with reference to the accompanying drawings.

[0042] The speech recognition method of this disclosure can be executed by the speech recognition device provided in this disclosure, which can be configured in an electronic device.

[0043] Figure 1 This is a schematic flowchart of a speech recognition method provided in an embodiment of the present disclosure.

[0044] like Figure 1 As shown, the speech recognition method may include the following steps:

[0045] Step 101: Obtain the spectrogram corresponding to the speech to be recognized.

[0046] The speech to be recognized can be any type of speech, such as a child's voice, a female voice, etc., and this disclosure does not limit it.

[0047] Optionally, after obtaining the to-be-recognized speech, the to-be-recognized speech can be processed to obtain a spectrum graph corresponding to the to-be-recognized speech. For example, Fourier transform processing can be performed, or other tools can be used for processing, to obtain a corresponding spectrum graph, and the like, which are not limited in the present disclosure.

[0048] It can be understood that the abscissa of the spectrum graph can be the time dimension, and the ordinate can be the frequency dimension, which are not limited in the present disclosure.

[0049] In step 102, the global feature corresponding to each feature point in the spectrum graph is determined according to the feature value of each feature point in the spectrum graph.

[0050] It can be understood that the feature value corresponding to each feature point in the spectrum graph can be the same or different, which are not limited in the present disclosure.

[0051] It can be understood that the global feature corresponding to the feature point in the spectrum graph can represent the weight of the feature point in the spectrum graph.

[0052] Optionally, the relationship between each two feature points in the spectrum graph can be determined first, then the correlation between each two feature points is determined according to the relationship between each two feature points, and then the global feature corresponding to each feature point can be determined by fusing the correlation corresponding to each feature point.

[0053] For example, there are four feature points in the spectrum graph, and the correlation between feature point 1 and feature point 2 is a 21 , the correlation between feature point 3 and feature point 1 is a 31 , and the correlation between feature point 4 and feature point 1 is a 41 . Then, a 21 , a 31 and a 41 are fused to determine the global feature corresponding to feature point 1 in the spectrum graph, which are not limited in the present disclosure.

[0054] Optionally, the correlation matrix between each feature point in the spectrum graph can also be determined according to the feature value of each feature point in the spectrum graph, and then the global feature is determined according to the correlation matrix and the feature value of each feature point.

[0055] It can be understood that the correlation between each two feature points can be determined according to the relationship between each two feature points in the spectrum graph, and then the correlation matrix between each feature point in the spectrum graph is determined according to the correlation between each two feature points.

[0056] For example, the size of the spectrum graph is 2*2, and there are four feature points, and the feature values of the feature points are respectively: By analyzing the relationship between each feature point in the spectrum graph and feature point 1, it can be determined that the correlation degree of feature point 1 and itself is a 21 , the correlation degree between feature point 2 and feature point 1: a 21 , the correlation degree between feature point 3 and feature point 1: a 31 , the correlation degree between feature point 4 and feature point 1: a 41 . Then, the correlation degree between each feature point and feature point 2 is determined: a 12 , a 22 , a 32 , a 42 ; the correlation degree between each feature point and feature point 3: a 13 , a 23 , a 33 , a 43 ; the correlation degree between each feature point and feature point 4: a 14 , a 24 , a 34 , a 44 . Then, according to the above correlation degrees, the correlation matrix between each feature point in the spectrum graph can be determined as:

[0057] Then, the correlation matrix between each feature point in the spectrum graph and the feature value of each feature point can be fused to determine the global feature:

[0058]

[0059] It should be noted that the above examples are only illustrative and cannot be used as a limitation on the size of the spectrum graph, the number of feature points, the feature value of each feature point, and the way of determining the correlation matrix between feature points in the embodiments of the present disclosure.

[0060] Step 103, determining the frequency domain feature of each frequency band in the spectrum graph according to the feature value of each feature point in the spectrum graph and the frequency band to which each feature point belongs.

[0061] Among them, there can be many ways to determine the frequency band. For example, the number of frequency bands and the frequency range corresponding to each frequency band can be set in advance; or the number of frequency bands can also be set in advance, and then the frequency range corresponding to each frequency band is determined according to the frequency value corresponding to each feature point in the spectrum graph, etc. The present disclosure does not limit this.

[0062] It can be understood that the spectrum diagram can contain the spectrum of multiple frequency bands. If the frequency domain feature corresponding to a frequency band is weak, the frequency band will be easily ignored in the identification process, that is, the feature information of the frequency band can be lost in the speech recognition process, which can cause the recognition result to be inaccurate. Therefore, in the embodiments of the present disclosure, in order to improve the accuracy of the recognition result, the spectrum diagram can be divided into frequency bands for feature extraction to enhance the frequency domain feature of the weak frequency band as much as possible, so that each frequency band in the spectrum diagram can occupy a certain proportion, thereby avoiding missing the feature information corresponding to the weak frequency band, so that the feature information is more complete and comprehensive, and the accuracy of the recognition result is improved.

[0063] In addition, there are various ways to determine the frequency domain feature of each frequency band in the spectrum diagram.

[0064] For example, the feature values of all feature points corresponding to each frequency band can be added first to obtain the feature mean value corresponding to each frequency band. Then, for the frequency band with a smaller feature mean value, a relatively larger weight can be assigned to obtain the frequency domain feature of the frequency band; for the frequency band with a larger feature mean value, a relatively smaller weight can be assigned to obtain the frequency domain feature of the frequency band. In this way, the weight of the frequency band with a weak feature mean value in the spectrum diagram can be increased, that is, the feature mean value of the weak frequency band is enhanced, so that the feature information of the frequency band will not be lost or missed in the subsequent processing process. Therefore, the completeness and comprehensiveness of the feature information are improved, and the accuracy of the recognition result is improved.

[0065] Step 104, determining the spectrum feature to be identified according to the global feature corresponding to the spectrum diagram and the frequency domain feature of each frequency band.

[0066] The global feature corresponding to the spectrum diagram and the frequency domain feature of the frequency band to which each feature point belongs can be fused to determine the spectrum feature to be identified, and the present disclosure does not limit this.

[0067] For example, the global feature corresponding to each feature point in the spectrum diagram can be multiplied by the frequency domain feature of the frequency band to which the feature point belongs to determine the spectrum feature to be identified; or the global feature corresponding to each feature point in the spectrum diagram can be weighted and fused with the frequency domain feature of the frequency band to which the feature point belongs to determine the spectrum feature to be identified. The present disclosure does not limit this.

[0068] It can be understood that in the embodiments of the present disclosure, the spectrum feature to be identified contains the frequency domain feature of each frequency band in the spectrum diagram, so that the spectrum feature to be identified determined is more comprehensive and reliable, and thus provides a basis for improving the accuracy of speech recognition.

[0069] Step 105, identifying the spectrum feature to be identified to determine the recognition result corresponding to the speech to be identified.

[0070] It should be noted that the spectrum features can be identified in any desirable manner to determine the recognition result of the speech to be identified, which is not limited in the present disclosure.

[0071] In the embodiments of the present disclosure, the spectrum graph corresponding to the speech to be identified can be acquired first, then the correlation matrix between each feature point in the spectrum graph can be determined according to the feature values of each feature point in the spectrum graph, then the frequency domain features of each frequency band in the spectrum graph can be determined according to the feature values of each feature point in the spectrum graph and the frequency band to which each feature point belongs, then the spectrum features to be identified can be determined according to the feature values of each feature point, the correlation matrix and the frequency domain features of each frequency band, and then the spectrum features are identified to determine the recognition result corresponding to the speech to be identified. Therefore, in the process of speech recognition, not only the relationship between each feature point in the spectrum graph is considered, but also the frequency domain features of each frequency band in the spectrum graph are considered, thereby improving the accuracy and integrity of the speech recognition result.

[0072] In the above embodiments, in the process of speech recognition, the relationship between each feature point in the spectrum graph and the frequency domain features of each frequency band are fully considered, thereby improving the accuracy and integrity of the speech recognition result when performing speech recognition. In actual implementation, the feature mean value corresponding to each frequency band can be determined first according to the feature values of each feature point in the spectrum graph and the frequency band to which each feature point belongs, then the feature values of the feature points are updated according to the feature mean value, and then the frequency domain features of the frequency band are determined, which will be described below in combination with Figure 2 The above process will be described in detail.

[0073] Figure 2 A flowchart of a speech recognition method provided by the embodiments of the present disclosure is shown in FIG. 1, which can include the following steps: Figure 2

[0074] Step 201, acquiring a spectrum graph corresponding to the speech to be identified.

[0075] Step 202, performing linear mapping on the spectrum graph based on a first mapping function and a second mapping function to obtain a first matrix and a second matrix after transformation.

[0076] The first mapping function and the second mapping function can be the same type of function or different types of function, and the parameters in the first mapping function and the parameters in the second mapping function can be the same or different, which is not limited in the present disclosure.

[0077] ​For example, the first mapping function and the second mapping function can be Gaussian functions with different parameters. If the size of the spectrum diagram is 6*6, the spectrum diagram is linearly mapped using the first mapping function, and the size of the transformed first matrix can be 36*1; the spectrum diagram is linearly mapped using the second mapping function, and the size of the transformed second matrix can be 1*36.

[0078] It should be noted that the above examples are only illustrative and cannot be used as a limitation on the size of the first mapping function, the second mapping function, the spectrum diagram, the first matrix, the second matrix, etc. in the embodiments of the present disclosure.

[0079] In step 203, the first matrix and the second matrix are multiplied to determine the association matrix.

[0080] For example, the size of the first matrix is 36*1, and the size of the second matrix is 1*36. If the two are multiplied, the association matrix can be obtained, and the size can be 36*36.

[0081] It should be noted that the above examples are only illustrative and cannot be used as a limitation on the size of the first matrix, the second matrix, the association matrix, etc. in the embodiments of the present disclosure.

[0082] In step 204, the global feature is determined according to the association matrix and the feature value of each feature point.

[0083] The association matrix and the feature value of each feature point can be fused to determine the global feature.

[0084] For example, the association matrix and the feature value of each feature point can be multiplied to determine the global feature, etc., which is not limited in the present disclosure.

[0085] In step 205, the feature mean value corresponding to each frequency band is determined according to the feature value of each feature point in the spectrum diagram and the frequency band to which each feature point belongs.

[0086] The frequency band to which each feature point belongs can be determined according to the frequency value corresponding to each feature point, and then the feature mean value corresponding to each frequency band can be determined according to the feature value of each feature point.

[0087] For example, feature point 1, feature point 2, and feature point 3 belong to frequency band 1, so the feature values of feature point 1, feature point 2, and feature point 3 can be added and averaged to obtain the feature mean value of frequency band 1; feature point 4 and feature point 5 belong to frequency band 2, so the feature values of feature point 4 and feature point 5 can be averaged to obtain the feature mean value of frequency band 2.

[0088] It should be noted that the above examples are only illustrative and cannot be used as a limitation on the feature points, feature values, frequency bands to which the feature points belong, and the manner of determining the feature mean value corresponding to the frequency band in the embodiments of the present disclosure.

[0089] In step 206, the feature value of each feature point is subtracted by the feature mean value corresponding to the frequency band to which the feature point belongs, to determine the updated feature value corresponding to each feature point.

[0090] For example, the feature value of feature point 1 is a1, the feature value of feature point 2 is a2, and the feature value of feature point 3 is a3. The frequency band to which feature point 1 belongs is frequency band 1, and the corresponding feature mean value is m1. The frequency bands to which feature point 2 and feature point 3 belong are frequency band 2, and the corresponding feature mean value is m2. Then, the updated feature value of feature point 1 can be determined as a1-m1, the updated feature value of feature point 2 is a2-m2, and the updated feature value of feature point 3 is a3-m2.

[0091] It should be noted that the above examples are only illustrative and cannot be used as a limitation on the feature points, feature values, frequency bands to which the feature points belong, and the manner of determining the feature mean value corresponding to the frequency band in the embodiments of the present disclosure.

[0092] In step 207, the updated maximum feature value in each frequency band is determined as the frequency domain feature of each frequency band.

[0093] Each frequency band can correspond to one feature point or multiple feature points, and the present disclosure does not limit this.

[0094] It can be understood that if a frequency band contains only one feature point, the updated feature value of the feature point can be determined as the frequency domain feature of the frequency band.

[0095] For example, frequency band 1 contains only feature point 5, and the updated feature value of feature point 5 can be determined as the frequency domain feature of frequency band 1.

[0096] It can be understood that if a frequency band contains multiple feature points, the updated feature values in the frequency band can be sorted according to their sizes, and the updated maximum feature value can be determined as the frequency domain feature of the frequency band.

[0097] For example, frequency band 2 contains feature point 2, feature point 3, and feature point 6, and the updated feature value of feature point 3 is the largest. Therefore, the updated feature value of feature point 3 can be determined as the frequency domain feature of frequency band 2.

[0098] It should be noted that the above examples are only illustrative and cannot be used as a limitation on the frequency bands, feature points, and updated feature values in the embodiments of the present disclosure.

[0099] Therefore, in the embodiments of the present disclosure, the feature values of each feature point can be updated by subtracting the feature mean corresponding to the frequency band to which the feature point belongs, so that the frequency domain features of the weak frequency band are enhanced, the loss or omission of feature information is avoided, the integrity and comprehensiveness of the feature information are ensured, and the accuracy of the recognition result is improved.

[0100] In step 208, the spectrum feature to be recognized is determined according to the global feature corresponding to the spectrum diagram and the frequency domain feature of each frequency band.

[0101] In step 209, the spectrum feature to be recognized is recognized to determine the recognition result corresponding to the speech to be processed.

[0102] In the embodiments of the present disclosure, the spectrum diagram corresponding to the speech to be recognized can be obtained first, then the spectrum diagram can be linearly mapped based on the first mapping function and the second mapping function to obtain the first matrix and the second matrix after transformation, and the first matrix and the second matrix are dot multiplied to determine the association matrix. Then, the feature mean corresponding to each frequency band can be determined according to the feature value of each feature point in the spectrum diagram and the frequency band to which each feature point belongs. Then, the updated feature value corresponding to each feature point can be determined by subtracting the feature mean corresponding to the frequency band to which the feature point belongs from the feature value of each feature point. Then, the frequency domain feature of each frequency band can be determined by the updated maximum feature value in each frequency band. The spectrum feature to be recognized is determined according to the feature value of each feature point, the association matrix and the frequency domain feature of each frequency band, and the spectrum feature is recognized to determine the recognition result corresponding to the speech. Therefore, in the process of speech recognition, not only the relationship between each feature point in the spectrum diagram is considered, but also the frequency domain feature of each frequency band in the spectrum diagram is considered, thereby improving the accuracy and integrity of the speech recognition result.

[0103] In the above embodiments, in the process of speech recognition, the feature mean corresponding to each frequency band is first determined according to the feature value of each feature point in the spectrum diagram and the frequency band to which each feature point belongs. Then, the feature value of the feature point is updated according to the feature mean, and then the frequency domain feature of the frequency band is determined. Then, the speech recognition result can be obtained by processing. In actual implementation, the global weight value corresponding to each feature point can be determined according to the association matrix first, then the feature value of each feature point is updated to obtain the updated feature value of each feature point, and then the processing is performed. The above process will be described in detail below. Figure 3 The above process will be described in detail.

[0104] In step 301, the spectrum diagram corresponding to the speech to be recognized is obtained.

[0105] In step 302, the association matrix between each feature point in the spectrum diagram is determined according to the feature value of each feature point in the spectrum diagram.

[0106] Step 303, according to the correlation matrix between each feature point in the spectrum diagram and the characteristic value of each feature point, determine the global feature.

[0107] Step 304, according to the characteristic value of each feature point in the spectrum diagram and the frequency band to which each feature point belongs, determine the frequency domain feature of each frequency band in the spectrum diagram.

[0108] Step 305, normalize the frequency domain feature of each frequency band to determine the frequency domain weight of each frequency band in the to-be-identified speech.

[0109] It can be understood that, in order to ensure the accuracy of the frequency domain weight, the frequency domain feature of each frequency band can be normalized first, and then the normalized frequency domain feature of each frequency band can be determined as the frequency domain weight.

[0110] For example, the frequency domain feature of frequency band 1 is a1, the frequency domain feature of frequency band 2 is b2, and the frequency domain feature of frequency band 3 is c3. If the normalized result is: frequency band 1 corresponds to d1, frequency band 2 corresponds to d2, and frequency band 3 corresponds to d3, the frequency domain weights corresponding to frequency band 1, frequency band 2 and frequency band 3 are: d1 / (d1+d2+d3), d2 / (d1+d2+d3), d3 / (d1+d2+d3) respectively.

[0111] It should be noted that the above example is only illustrative and cannot be used as a limitation on the frequency bands, the frequency domain features of the frequency bands, and the way of determining the frequency domain weight of each frequency band in the embodiments of the present disclosure.

[0112] Step 306, according to the frequency domain weight of each frequency band and the global feature corresponding to the spectrum diagram, determine the to-be-identified spectrum feature.

[0113] It can be understood that the frequency domain weight of each frequency band and the global feature of each feature point in the spectrum diagram can be fused to determine the to-be-identified spectrum feature.

[0114] For example, the frequency domain weight of each frequency band can be multiplied by the global feature of each feature point in the spectrum diagram, and then fused, and the result obtained is the to-be-identified spectrum feature.

[0115] For example, the global feature of feature point 1 in the spectrum diagram is b1, the frequency band to which feature point 1 belongs is frequency band 1, and the frequency domain weight of frequency band 1 is c1. Then the fusion result of feature point 1 is b1*c1. If the global feature of each feature point in the spectrum diagram is multiplied by the frequency domain weight to which it belongs, the fusion result of each feature point can be obtained, and the fusion result of all feature points in the spectrum diagram is the to-be-identified spectrum feature.

[0116] It should be noted that the above examples are only illustrative and cannot be used as a limitation on the feature points, the updated feature values, the frequency domain weights, etc. in the embodiments of the present disclosure.

[0117] In step 307, the spectral features are recognized to determine the recognition result corresponding to the speech to be processed.

[0118] In the embodiments of the present disclosure, the spectral graph corresponding to the speech to be recognized can be acquired, then the correlation matrix between each feature point in the spectral graph is determined according to the feature value of each feature point in the spectral graph, the global feature of each feature point is determined according to the correlation matrix and the feature value of each feature point, the frequency domain feature of each frequency band in the spectral graph is determined according to the feature value of each feature point in the spectral graph and the frequency band to which each feature point belongs, the frequency domain feature of each frequency band is normalized to determine the frequency domain weight of each frequency band in the speech to be recognized, the spectral feature to be recognized is determined according to the frequency domain weight of each frequency band and the global feature of each feature point in the spectral graph, and then the spectral feature is recognized to determine the recognition result corresponding to the speech to be processed. Therefore, in the process of speech recognition, not only the relationship between each feature point in the spectral graph is considered, but also the frequency domain feature of each frequency band in the spectral graph is considered, thereby improving the accuracy and integrity of the speech recognition result.

[0119] To implement the above-mentioned embodiments, the present disclosure further provides a speech recognition model.

[0120] Figure 4 A structural schematic diagram of a speech recognition model provided by the embodiments of the present disclosure is shown.

[0121] As shown in Figure 4 , the speech recognition model can include N speech feature extraction modules and a speech recognition module connected in sequence, where N is a positive integer greater than 1.

[0122] Each speech feature extraction module can include a convolutional network, a global attention network, and a frequency domain attention network.

[0123] It can be understood that the number of channels corresponding to the N speech feature extraction modules can be the same or different, and the present disclosure does not limit this.

[0124] The number of channels can represent the granularity size of the convolution kernel, so that when processing, the speech feature extraction modules with different numbers of channels, i.e., the convolution kernels with different granularities, are used, so that the extracted speech features are more comprehensive and reliable.

[0125] It can be understood that a convolutional neural network (CNN) can be used to perform convolution processing on the obtained speech features to obtain more robust speech features.

[0126] The convolution layer of the convolutional neural network can include a convolution (CONV), a batch normalization (BN), a rectified linear units (ReLU) activation function, or other network structures, which are not limited in the present disclosure.

[0127] In addition, the global attention network can be used to process the new speech features to output global features of the new speech features.

[0128] In addition, the frequency domain attention network can be used to process the new speech features to determine the frequency domain features of each frequency band in the new speech features.

[0129] In addition, the speech recognition module can be used to recognize the global features and the frequency domain features output by the Nth speech feature extraction module to output a recognition result.

[0130] The following takes a speech feature extraction module as an example, and combines the above description of the speech recognition model provided by the present disclosure to describe the speech recognition process. Figure 4A The structure of the speech recognition model provided by the present disclosure and the speech recognition process are described in detail.

[0131] Figure 4A A structure diagram of a speech recognition model provided by the present disclosure.

[0132] As shown in the figure, first, the speech to be recognized can be processed to obtain corresponding speech features. Figure 4A

[0133] Then, the speech features can be input into the convolutional neural network to obtain more robust new speech features.

[0134] The convolutional neural network can include a convolution layer (Conv layer), a BN layer, and a ReLU layer, which are not limited in the present disclosure.

[0135] Then, the new speech features can be input into the global attention network to process the feature values of each feature point in the new speech features by using a first mapping function and a second mapping function in the global attention network to obtain a first matrix and a second matrix after transformation, and then point-multiply the first matrix and the second matrix to determine the correlation matrix between each feature point in the new speech features.

[0136] ​In addition, the third mapping function can be the same as or different from the first mapping function and the second mapping function, and the present disclosure does not limit this.

[0137] It can be understood that the new speech feature can be enhanced according to its own characteristics through the third mapping function to obtain a corresponding third matrix, and the present disclosure does not limit this.

[0138] Then, the global feature of the new speech feature is obtained by fusing the correlation matrix after the softmax processing and the third matrix.

[0139] In addition, the new speech feature can be input into the frequency domain attention network, so that the frequency domain attention network can process the new speech feature to determine the frequency domain feature of each frequency band in the new speech feature.

[0140] For example, there are N CNN networks in the frequency domain attention network, where N is any positive number. The new speech feature can be sent into the N CNN networks according to the different frequencies, and after processing, the feature value of each feature point in the new speech feature and the frequency band to which each feature point belongs can be obtained. Then, the feature mean corresponding to each frequency band is determined, and then the feature value of each feature point is subtracted from the feature mean corresponding to the frequency band to which it belongs, and the updated feature value corresponding to each feature point is determined. Then, the updated feature value is sent into the max pooling layer according to the frequency band to which the feature point belongs, so as to determine the frequency domain feature of each frequency band.

[0141] Then, the global feature of the new speech feature and the frequency domain feature of each frequency band in the new speech feature can be fused, and the fused result is input into the speech recognition module.

[0142] Optionally, after the frequency domain feature of each frequency band is determined, the frequency domain feature of each frequency band can also be sent into the fully connected layer (FC), and then through the normalization processing, the frequency domain weight of each frequency band in the new speech feature can be obtained.

[0143] The above process can be represented by the following formula (1):

[0144] M fre = softmax(FC(maxpooling(CNN(x)))) (1) where M fre is the frequency domain weight, softmax is the output layer, FC is the fully connected layer, maxpooling is the max pooling layer, CNN is the convolutional neural network, and x is the new speech feature.

[0145] Optionally, in actual implementation, after the frequency domain weight is determined, the frequency domain features can be fused according to the frequency domain weight.

[0146] For example, the frequency domain weight can be fused with the third matrix first to obtain updated frequency domain features:

[0147] x' z w' z =x z w z *M fre (2)

[0148] wherein x′ z w′ z is the updated frequency domain features, x z w z is the third matrix, and M fre is the frequency domain weight.

[0149] Then, the global features and the updated frequency domain features can be fused to obtain the global features and the frequency domain features to be identified:

[0150] X=softmax(x i w i w j x j )*(x' z w' z ) (3)

[0151] wherein X is the global features and the frequency domain features to be identified, softmax(x i w i w j x j ) is the global features, and x′ z w′ z is the updated frequency domain features.

[0152] Then, the speech recognition module can identify the global features and the frequency domain features output by the speech feature extraction module to output the recognition result.

[0153] For example, the frequency domain features of each frequency band in the new speech features and the global features of the new speech features can be input into the speech recognition module, and then the recognition result can be output after processing by the speech recognition module.

[0154] Optionally, the frequency domain features of each frequency band in the new speech features and the global features of the new speech features can be fused first, and then input into the speech recognition module.

[0155] It should be noted that the above examples are only illustrative and cannot be regarded as a limitation on the input of the speech recognition module in the embodiments of the present disclosure.

[0156] Optionally, the speech recognition module can comprise a connectionist temporal classification (CTC) based on a neural network, so that the CTC is used to recognize the global features and the frequency domain features output by the speech feature extraction module, and then output the recognition result, etc., which is not limited in the present disclosure.

[0157] The speech recognition model provided by the embodiments of the present disclosure can comprise N speech feature extraction modules and a speech recognition module connected in sequence, so that the acquired speech features are input to the speech feature extraction module, and after being processed by the N speech feature extraction modules, they are input to the speech recognition module, and then the speech recognition result can be output. Therefore, in the process of speech recognition, not only the relationship between each feature point in the spectrogram is considered, but also the frequency domain features of each frequency band in the spectrogram are considered, thereby improving the accuracy and integrity of the speech recognition result.

[0158] In order to realize the above-mentioned embodiments, the present disclosure further provides a speech recognition device.

[0159] Figure 5 A structural schematic diagram of a speech recognition device provided by the embodiments of the present disclosure.

[0160] As shown in Figure 5 The speech recognition device 500 comprises an acquisition module 510, a first determination module 520, a second determination module 530, a third determination module 540, and a recognition module 550.

[0161] The acquisition module 510 is configured to acquire a spectrogram corresponding to a speech to be recognized.

[0162] The first determination module 520 is configured to determine global features corresponding to each feature point in the spectrogram according to feature values of each feature point in the spectrogram.

[0163] The second determination module 530 is configured to determine frequency domain features of each frequency band in the spectrogram according to the feature values of each feature point in the spectrogram and the frequency band to which each feature point belongs.

[0164] The third determination module 540 is configured to determine the spectrogram features to be recognized according to the global features corresponding to the spectrogram and the frequency domain features of each frequency band.

[0165] The recognition module 550 is configured to recognize the spectrogram features to determine a recognition result corresponding to the speech.

[0166] Optionally, the first determination module 520 comprises:

[0167] The first determining unit is configured to determine a correlation matrix between each feature point in the spectrum diagram according to the feature values of the feature points in the spectrum diagram.

[0168] The second determining unit is configured to determine the global feature according to the correlation matrix and the feature values of each feature point.

[0169] Optionally, the first determining unit is specifically configured to:

[0170] linearly map the spectrum diagram based on a first mapping function and a second mapping function to obtain a first matrix and a second matrix after transformation;

[0171] point-multiply the first matrix and the second matrix to determine the correlation matrix.

[0172] Optionally, the second determining module 530 is specifically configured to:

[0173] determine a feature mean value corresponding to each frequency band according to the feature values of each feature point in the spectrum diagram and the frequency band to which each feature point belongs;

[0174] subtract the feature mean value corresponding to the frequency band to which each feature point belongs from the feature value of each feature point to determine an updated feature value corresponding to each feature point;

[0175] determine a frequency domain feature of each frequency band as the maximum updated feature value in the frequency band.

[0176] Optionally, the third determining module 540 is configured to:

[0177] normalize the frequency domain feature of each frequency band to determine a frequency domain weight of the frequency domain feature of each frequency band in the to-be-recognized speech;

[0178] determine the to-be-recognized spectrum feature according to the frequency domain weight of each frequency band and the updated feature value of each feature point.

[0179] The functions and specific implementation principles of the above modules in the embodiments of the present disclosure can be referred to the above method embodiments, which will not be described here.

[0180] The speech recognition device of this disclosure first acquires the spectrogram corresponding to the speech to be recognized. Then, based on the feature values ​​of each feature point in the spectrogram, it determines the correlation matrix between the feature points in the spectrogram. Next, based on the feature values ​​of each feature point and the frequency band to which each feature point belongs, it determines the frequency domain features of each frequency band in the spectrogram. Finally, based on the feature values ​​of each feature point, the correlation matrix, and the frequency domain features of each frequency band, it determines the spectral features to be recognized. Finally, it recognizes the spectral features to determine the recognition result corresponding to the speech to be recognized. Therefore, in the process of speech recognition, not only the relationships between the feature points in the spectrogram are considered, but also the frequency domain features of each frequency band in the spectrogram, thereby improving the accuracy and completeness of the speech recognition results.

[0181] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0182] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0183] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded from storage unit 608 into random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0184] A plurality of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunication networks.

[0185] The computing unit 601 can be various general and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 601 performs various methods and processes described above, such as the speech recognition method. For example, in some embodiments, the speech recognition method can be implemented as a computer software program, which is tangibly embodied in a machine-readable medium, such as the storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded onto the RAM 603 and executed by the computing unit 601, one or more steps of the speech recognition method described above can be performed. Alternatively, in other embodiments, the computing unit 601 can be configured to perform the speech recognition method by any other appropriate means, such as by means of firmware.

[0186] The various implementations of the systems and techniques described above herein can be realized in a digital electronic circuit system, an integrated circuit system, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), an application specific standard product (ASSP), a system on a chip system (SOC), a complex programmable logic device (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0187] Program code for carrying out methods of the present disclosure can be written in any combination of one or more programming languages. The program code can be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the program code, when executed by the processor or controller, produces the functions / operations specified in the flowcharts and / or block diagrams. The program code can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package, or entirely on a remote machine or server.

[0188] In the context of the present disclosure, a machine-readable medium can be a tangible medium that contains or stores a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include but is not limited to an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of the machine-readable storage medium will include one or more lines of electrical connections, portable computer disks, hard disk drives, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or Flash memory), optical fibers, portable compact disc read-only memories (CD-ROMs), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0189] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0190] The systems and techniques described herein can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0191] The computer system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service ("Virtual Private Server", or simply "VPS"). The server can also be a server of a distributed system, or a server combined with a blockchain.

[0192] The technical solution of the present disclosure can first acquire a spectrum graph corresponding to the to-be-identified voice, then determine an association matrix between each feature point in the spectrum graph according to a feature value of each feature point in the spectrum graph, then determine a frequency domain feature of each frequency band in the spectrum graph according to the feature value of each feature point in the spectrum graph and a frequency band to which each feature point belongs, then determine the to-be-identified spectrum feature according to the feature value of each feature point, the association matrix, and the frequency domain feature of each frequency band, and then identify the spectrum feature to determine an identification result corresponding to the to-be-identified voice. Thus, in the process of voice identification, not only the relationship between each feature point in the spectrum graph is considered, but also the frequency domain feature of each frequency band in the spectrum graph is considered, thereby improving the accuracy and integrity of the voice identification result.

[0193] It should be understood that the various forms of flow shown above can be reordered, added to, or deleted from, with steps. For example, each step described in the present disclosure can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution disclosed in the present disclosure can be achieved, and the present disclosure is not limited herein.

[0194] The above detailed description does not limit the scope of the disclosure. Various modifications, combinations, sub-combinations and alternatives can be made to the detailed description. Any modification, equivalent replacement and improvement etc. made within the spirit and principle of the disclosure shall be included in the scope of the disclosure.

Claims

1. A speech recognition method, characterized by, The method comprises: obtaining a spectrum graph corresponding to the voice to be recognized; determining a global feature corresponding to each feature point in the spectrum graph according to a feature value of each feature point in the spectrum graph; determining a frequency domain feature of each frequency band in the spectrum graph according to the feature value of each feature point in the spectrum graph and a frequency band to which each feature point belongs; determining a spectrum feature to be recognized according to the global feature corresponding to the spectrum graph and the frequency domain feature of each frequency band; recognizing the spectrum feature to determine a recognition result corresponding to the voice; the determining of the global feature corresponding to each feature point in the spectrum graph according to the feature value of each feature point in the spectrum graph comprises: determining a correlation matrix between each feature point in the spectrum graph according to the feature value of each feature point in the spectrum graph; determining the global feature according to the correlation matrix and the feature value of each feature point; the determining of the frequency domain feature of each frequency band in the spectrum graph according to the feature value of each feature point in the spectrum graph and the frequency band to which each feature point belongs comprises: determining a feature mean value corresponding to each frequency band according to the feature value of each feature point in the spectrum graph and the frequency band to which each feature point belongs; subtracting the feature mean value corresponding to the frequency band to which each feature point belongs from the feature value of each feature point to determine an updated feature value corresponding to each feature point; determining the frequency domain feature of each frequency band by taking the updated maximum feature value in each frequency band.

2. The method of claim 1, wherein, the determining of the correlation matrix between each feature point in the spectrum graph according to the feature value of each feature point in the spectrum graph comprises: linearly mapping the spectrum graph based on a first mapping function and a second mapping function to obtain a first matrix and a second matrix after transformation respectively; point-multiplying the first matrix and the second matrix to determine the correlation matrix.

3. The method of claim 1, wherein, the determining of the spectrum feature to be recognized according to the global feature and the frequency domain feature of each frequency band comprises: normalizing the frequency domain feature of each frequency band to determine a frequency domain weight of the frequency domain feature of each frequency band in the voice to be recognized; determining the spectrum feature to be recognized according to the frequency domain weight of each frequency band and the global feature.

4. A speech recognition model, employing the method of any one of claims 1 to 3, characterized in that, The method comprises: N voice feature extraction modules and a voice recognition module connected in sequence, wherein N is a positive integer greater than 1; each voice feature extraction module comprises a convolution network, a global attention network and a frequency domain attention network; the convolution network is configured to perform convolution processing on the obtained voice feature to obtain a new voice feature; the global attention network is configured to process the new voice feature to output a global feature of the new voice feature; the frequency domain attention network is configured to process the new voice feature to determine a frequency domain feature of each frequency band in the new voice feature; the voice recognition module is configured to recognize the global feature and the frequency domain feature output by the Nth voice feature extraction module to output a recognition result.

5. The model of claim 4, wherein, The number of channels corresponding to the N voice feature extraction modules is different.

6. A speech recognition apparatus employing the method of any one of claims 1 to 3, the apparatus comprising: an obtaining module configured to obtain a spectrogram corresponding to a speech to be recognized; a first determining module configured to determine a global feature corresponding to each feature point in the spectrogram according to a feature value of each feature point in the spectrogram; a second determining module configured to determine a frequency domain feature of each frequency band in the spectrogram according to the feature value of each feature point in the spectrogram and a frequency band to which each feature point belongs; a third determining module configured to determine a spectrum feature to be recognized according to the global feature and the frequency domain feature of each frequency band in the spectrogram; a recognition module configured to recognize the spectrum feature to determine a recognition result corresponding to the speech.

7. The apparatus of claim 6, wherein, The first determining module comprises: a first determining unit configured to determine a correlation matrix between each feature point in the spectrogram according to a feature value of each feature point in the spectrogram; a second determining unit configured to determine the global feature according to the correlation matrix and the feature value of each feature point.

8. The apparatus of claim 7, wherein, The first determining unit is specifically configured to: linearly map the spectrogram based on a first mapping function and a second mapping function to obtain a first matrix and a second matrix after transformation, respectively; point-multiply the first matrix and the second matrix to determine the correlation matrix.

9. The apparatus of claim 6, wherein, The second determining module is specifically configured to: determine a feature mean value corresponding to each frequency band according to the feature value of each feature point in the spectrogram and the frequency band to which each feature point belongs; subtract the feature mean value corresponding to the frequency band to which each feature point belongs from the feature value of each feature point to determine an updated feature value corresponding to each feature point; determine the updated maximum feature value in each frequency band as the frequency domain feature of each frequency band.

10. The apparatus of claim 6, wherein, The third determining module is configured to: normalize the frequency domain feature of each frequency band to determine a frequency domain weight of the frequency domain feature of each frequency band in the speech to be recognized; determine the spectrum feature to be recognized according to the frequency domain weight of each frequency band and the global feature.

11. An electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-3. The computer instructions are used to enable the computer to perform the method of any one of claims 1-3.

12. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, 13. A computer program product comprising a computer program which, when executed by a processor, implements the method of any one of claims 1-3. ​

Images