Speech recognition method and computer storage medium

Abstract

Embodiments of the present application provide a speech recognition method and a computer storage medium. The speech recognition method comprises: performing semantic feature extraction and attribute feature extraction on audio feature data of a speech to be recognized, wherein the attribute feature extraction comprises at least accent feature extraction; obtaining character information of the speech to be recognized according to a result of the semantic feature extraction; generating an adaptive vector corresponding to each speech character according to at least an accent feature vector of the speech to be recognized obtained by the attribute feature extraction and an initial vector matched with each speech character determined according to the character information; obtaining an attribute feature corresponding to the adaptive vector from an attribute feature library storing speech attribute features; and obtaining a recognition result of each speech character according to the attribute feature and the initial vector. The method can adapt to the recognition of speech with different accents and improve the recognition accuracy.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a speech recognition method and a computer storage medium. Background Art

[0002] With the development of deep learning technology, speech recognition based on neural network models has been widely used in various scenarios such as human-computer interaction, meetings, and court records.

[0003] Existing speech recognition technology often uses standard training samples, such as standard Mandarin speech samples, when training neural network models to ensure that the trained neural network models can effectively recognize Chinese speech. However, in practice, although Mandarin is the standard for Chinese, local dialects vary. Therefore, people in different regions may not speak standard Mandarin, but rather Mandarin with a local accent. This can lead to deviations in speech recognition and inaccurate results. Similar issues exist for other languages ​​as well.

[0004] Therefore, how to provide a speech recognition solution to perform accurate speech recognition in the aforementioned scenarios has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, an embodiment of the present application provides a speech recognition solution to at least partially solve the above problems.

[0006] According to a first aspect of an embodiment of the present application, a speech recognition method is provided, comprising: performing semantic feature extraction and attribute feature extraction on audio feature data of a speech to be recognized, respectively, wherein the attribute feature extraction includes at least accent feature extraction; obtaining character information of the speech to be recognized based on a result of the semantic feature extraction; generating an adaptive vector corresponding to each speech character based on at least an accent feature vector of the speech to be recognized obtained by the attribute feature extraction and an initial vector matching each speech character determined based on the character information; obtaining attribute features corresponding to the adaptive vector from an attribute feature library storing speech attribute features; and obtaining a recognition result for each speech character based on the attribute features and the initial vector.

[0007] According to the second aspect of the embodiment of the present application, a speech recognition method is provided, including: receiving a speech to be recognized, and sending the speech to be recognized to the cloud by calling an interface, so that the cloud performs speech recognition on the speech to be recognized according to the method as described above; and receiving the speech recognition result returned by the cloud.

[0008] According to the third aspect of the embodiment of the present application, a speech recognition method is provided, including: collecting conference speech in real time, and sending the conference speech to the cloud by calling an interface, so that the cloud performs speech recognition on the conference speech according to the speech recognition method as described above; receiving the speech recognition result returned by the cloud, and displaying the meeting record based on the speech recognition result.

[0009] According to the fourth aspect of the embodiment of the present application, a speech recognition method is provided, including: performing semantic feature extraction on audio feature data of a speech to be recognized through a semantic extraction part of an autoregressive speech recognition model; generating an adaptive vector carrying the accent features of the speech to be recognized based on the speech to be recognized through an adaptive part of the autoregressive speech recognition model and obtaining attribute features corresponding to the adaptive vector; and performing recognition output of the speech to be recognized according to the attribute features through a recognition output part of the autoregressive speech recognition model.

[0010] According to the fifth aspect of the embodiment of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the speech recognition method described in any one of the first to fourth aspects.

[0011] According to a sixth aspect of an embodiment of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the speech recognition method as described in any one of the first to fourth aspects is implemented.

[0012] According to the speech recognition solution provided in the embodiment of the present application, the adaptability in the speech recognition process is improved by extracting an independent adaptive vector for each speech character and determining the corresponding attribute features based on the adaptive vector corresponding to the speech character. In the case of speech to be recognized that can adapt to different accents, there is no need to additionally train the classifier for accent classification, which saves training costs and avoids the problems of reduced adaptability and inaccurate recognition caused by inaccurate classification of the classifier. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0014] Figure 1AThis is a schematic diagram of the structure of the autoregressive speech recognition module of Example 1 of the present application;

[0015] Figure 1B This is a flowchart of a speech recognition method according to Embodiment 1 of the present application;

[0016] Figure 2 This is a flowchart of a speech recognition method according to the second embodiment of the present application;

[0017] Figure 3A This is a flowchart of a speech recognition method according to the third embodiment of the present application;

[0018] Figure 3B for Figure 3A A schematic diagram of a speech recognition scenario in the third embodiment is shown;

[0019] Figure 4A This is a flowchart of a speech recognition method according to the fourth embodiment of the present application;

[0020] Figure 4B A schematic diagram of a display interface for displaying speech recognition results;

[0021] Figure 4C A schematic diagram of another display interface for displaying speech recognition results;

[0022] Figure 5 This is a structural block diagram of a speech recognition device according to a fifth embodiment of the present application;

[0023] Figure 6 This is a structural block diagram of a speech recognition device according to a sixth embodiment of the present application;

[0024] Figure 7 This is a structural block diagram of a speech recognition device according to a seventh embodiment of the present application;

[0025] Figure 8 This is a structural diagram of an electronic device according to Example 8 of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present application.

[0027] The specific implementation of the embodiment of the present application is further explained below in conjunction with the accompanying drawings of the embodiment of the present application.

[0028] Example 1

[0029] In this embodiment, the audio feature data of the speech to be recognized is processed by an autoregressive speech recognition model to obtain a more accurate speech to be recognized that can adapt to different accents or language habits, thereby improving the recognition accuracy.

[0030] To facilitate explanation and understanding, before explaining the implementation process of the speech recognition method, the structure of the autoregressive speech recognition model is briefly described as follows:

[0031] like Figure 1A As shown, the autoregressive speech recognition model includes a semantic extraction component, an adaptive component, and a recognition output component. Taking this model as an example, a speech recognition method based on this model can be implemented as follows: the semantic extraction component of the autoregressive speech recognition model extracts semantic features from the audio feature data of the speech to be recognized; the adaptive component of the autoregressive speech recognition model generates an adaptive vector carrying the accent features of the speech to be recognized based on the speech to be recognized and obtains attribute features corresponding to the adaptive vector; and the recognition output component of the autoregressive speech recognition model outputs recognition of the speech to be recognized based on the attribute features.

[0032] The following further describes the various parts of the autoregressive speech recognition model.

[0033] Among them, the semantic extraction part is used to extract semantic features from the audio feature data of the speech to be recognized to obtain the results of the semantic feature extraction. According to different needs, the semantic feature extraction part can include different components. For example, the semantic feature extraction part includes a feature encoder, which can be a SAN-M encoder or other encoder capable of extracting semantic features from audio feature data, without limitation. The semantic features extracted by the semantic extraction part will be used in the subsequent recognition output part to obtain the vector (initial vector) corresponding to the voice characters of the speech to be recognized.

[0034] The adaptive part is used to generate an adaptive vector carrying the accent features of the speech to be recognized based on the speech to be recognized and obtain attribute features corresponding to the adaptive vector. For example, the adaptive part may include an adaptive encoder, an adaptive decoder, a self-attention layer, and a label synchronization layer.

[0035] The adaptive encoder is used to extract attribute features from the audio feature data of the speech to be recognized and output a corresponding encoding vector (attribute feature vector). The encoding vector carries at least the accent feature vector and, of course, may also include other information such as speaker identification as needed, without limitation.

[0036] The adaptive decoder is used to decode the encoding vector output by the adaptive encoder and receive the initial vector output by the feature decoder in the recognition output part, and generate an adaptive vector corresponding to each voice character according to the decoding result and the initial vector.

[0037] The self-attention layer can be a multi-head self-attention layer (Multi-head Attention), which is used to match attribute features corresponding to the adaptive vector output by the adaptive decoder from an attribute feature library (GST) storing speech attribute features through self-attention calculation. The attribute feature library pre-stores attribute features of audio with different accents, including accent feature vectors. However, this is not limited to this, and it can also include speaker identification, etc. The accent feature vector can be obtained by machine learning of audio with different accents using an appropriate neural network model.

[0038] The label synchronization embedding layer generates a label synchronization vector based on the matched attribute features and inputs it into the feature decoder of the recognition output. For example, it can convert the dimension of the attribute features to the dimension required by the label synchronization vector, thereby aligning the attribute features into a unified feature space.

[0039] Because the adaptive part extracts an adaptive vector for each spoken character in an unsupervised manner through an adaptive encoder, adaptive decoder, self-attention layer, and label synchronization layer, it avoids the problem of inaccurate classification of audio feature data using a Dominant classifier, which can lead to reduced adaptive performance. Furthermore, because it doesn't require a Dominant classifier, there's no need for pre-training, achieving plug-and-play compatibility and greater compatibility with audio feature data with new accents or language habits.

[0040] The recognition output section is used to output the recognition of the speech to be recognized based on the attribute features corresponding to the adaptive vector. The recognition output section includes a feature decoder (FSMN decoder), which is used to output the recognition result of each speech character based on the initial vector and the label synchronization vector of each speech character.

[0041] In addition, it should be noted that the aforementioned feature encoder and feature decoder can adopt transformer-based ASR. Of course, other appropriate network structures can also be used, and there is no limitation on this.

[0042] The following describes the speech recognition method in combination with the autoregressive speech recognition model:

[0043] like Figure 1BAs shown, the method includes the following steps:

[0044] Step S102: performing semantic feature extraction and attribute feature extraction on the audio feature data of the speech to be recognized.

[0045] The audio feature data of the speech to be recognized may be a Mel-spectrum, and of course it may also be other appropriate feature data, which is not limited to this.

[0046] The audio feature data is input into the feature encoder of the autoregressive speech recognition model, and semantic features are extracted using the feature encoder to obtain the semantic feature extraction results (also known as semantic feature encoding). In addition, the audio feature data is input into the adaptive encoder of the autoregressive speech recognition model, and attribute features are extracted using the adaptive encoder. The attribute feature extraction includes at least accent feature extraction to obtain an accent feature vector, which is used to characterize the accent of the speech to be recognized.

[0047] In this embodiment, since the accent feature vector is obtained by attribute extraction to represent the accent, there is no need to use a domain classifier to classify the accent of the speech to be recognized, thereby avoiding the decrease in the adaptability of speech recognition due to inaccurate classification of the domain classifier, thereby improving the recognition accuracy and adaptability.

[0048] Step S104: obtaining character information of the speech to be recognized according to the result of the semantic feature extraction.

[0049] The character information carries a token, which indicates the audio segment corresponding to each voice character in the speech to be recognized. The feature decoder can determine the number of voice characters contained in the speech to be recognized, the audio segment corresponding to each voice character, etc. by processing the results of semantic feature extraction. In addition, the character information also includes the language information corresponding to each voice character, such as <cn>The tag indicates that the phonetic character is Chinese, with <en>The tag indicates that the phonetic character is English, etc.

[0050] By carrying tokens and corresponding language labels in character information, the adaptability of the recognition process can be further improved, thereby improving recognition accuracy.

[0051] Step S106: generating an adaptive vector corresponding to each voice character based on the accent feature vector of the voice to be recognized obtained by extracting the attribute features and the initial vector matched with each voice character determined based on the character information.

[0052] In this embodiment, in order to adapt to speech to be recognized with different accents and ensure that accurate text can be recognized for speech to be recognized with different accents, an adaptive vector expressing language attribute features is generated for each speech character, so that fine-grained (i.e., speech character granularity) attribute feature extraction can be achieved.

[0053] The initial vector that matches the phonetic character can be the character vector of the recognized phonetic character. For example, if the character information indicates that the to-be-recognized speech includes four phonetic characters, A, B, C, and D, then when determining the adaptive vector for phonetic character B, the initial vector that matches phonetic character B can be the character vector of the already-obtained phonetic character A.

[0054] When determining the adaptive vector for each spoken character, the system not only uses the accent feature vector but also combines the initial vector corresponding to each spoken character in the character information. For example, the initial vector and the accent feature vector can be fused to obtain the adaptive vector. This allows the adaptive vector to integrate both the accent features and the contextual information of the speech being recognized.

[0055] Step S108: Acquire the attribute feature corresponding to the adaptive vector from an attribute feature library storing speech attribute features.

[0056] The attribute feature library (e.g., GST) pre-stores attribute features for audio with different accents. The speech attribute features include, but are not limited to, the corresponding accent feature vector of the audio. They may also include a speaker identity. The accent feature vector may include, but is not limited to, accent and pronunciation features.

[0057] By matching the adaptive vector of the phonetic character with the attribute features in the attribute feature library, accurate attribute features of the phonetic character can be obtained, thereby obtaining more accurate attribute features of the current phonetic character.

[0058] Step S110: obtaining a recognition result of each phonetic character according to the attribute feature and the initial vector.

[0059] For each voice character in the speech to be recognized, the attribute features and the initial vector (that is, the character vector of the previously predicted voice character) are fused in the feature decoder to determine its corresponding character vector, and then the text corresponding to the voice character is determined as the recognition result based on the character vector.

[0060] The speech recognition method of this embodiment improves the adaptability in the speech recognition process by extracting an independent adaptive vector for each speech character and determining the corresponding attribute features based on the adaptive vector corresponding to the speech character. In the case of being able to adapt to the speech to be recognized with different accents, there is no need to additionally train the classifier for accent classification, which saves training costs and avoids the problems of reduced adaptability and inaccurate recognition caused by inaccurate classification of the classifier.

[0061] The method of this embodiment can be executed by any appropriate electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as mobile phones and PADs with high software and hardware performance), and PCs.

[0062] Example 2

[0063] Reference Figure 2 , shows a schematic flow chart of the steps of the speech recognition method of Example 2 of the present application.

[0064] Step S202: performing semantic feature extraction and attribute feature extraction on the audio feature data of the speech to be recognized.

[0065] In one example, the audio feature data may be a mel spectrum obtained by processing the speech to be recognized. The audio feature data may be input into a feature encoder to extract semantic features, thereby obtaining semantic feature codes as the result of the semantic feature extraction.

[0066] The Mel spectrum is input into the adaptive encoder for attribute feature extraction. In this embodiment, the attribute feature extraction includes accent feature extraction. The accent feature extraction can obtain an accent feature vector, which includes accent, pronunciation characteristics, etc. In addition, the attribute feature extraction can also include speaker identification extraction. The speaker identification can be obtained through speaker identification extraction, which can be represented in the form of a vector. By extracting the speaker identification, the information contained in the subsequent adaptive vector is richer and more targeted, which helps to improve the accuracy of recognition.

[0067] Step S204: obtaining character information of the speech to be recognized based on the result of the semantic feature extraction.

[0068] In one feasible method, the result of semantic feature extraction output by the feature encoder is input into the feature decoder. The feature decoder can output character information by processing the result of semantic feature extraction. The character information includes one or more tokens, which indicate the phonetic characters contained in the speech to be recognized and the audio segment corresponding to the phonetic characters through tokens, so as to facilitate subsequent prediction of phonetic characters.

[0069] Optionally, in order to facilitate subsequent autoregressive prediction, a start symbol can be added before the first token of the character information. The start symbol can be any appropriate identifier, such as <sos>Similarly, a terminator can be added after the last token. The terminator can be any appropriate identifier, such as <eos>.

[0070] In addition, the character information can also carry the language information corresponding to the phonetic characters. The language information can be <cn> 、 <en>The labels indicate the language of the voice characters. For example, if the voice characters corresponding to the audio to be recognized are "开AA", the labels of the language information corresponding to the voice characters "打" and "开" are <cn>, the language information label corresponding to the phonetic character "A" is <en>.

[0071] Step S206: generating an adaptive vector corresponding to each voice character based on the accent feature vector of the voice to be recognized obtained by extracting the attribute features and the initial vector matched with each voice character determined based on the character information.

[0072] The initial vector for a phonetic character can be the character vector corresponding to the phonetic character output by the feature decoder during the previous decoding process. For example, in the speech to be recognized, "开AA", the initial vector corresponding to the phonetic character "打" can be the character vector corresponding to the preset start symbol. The initial vector corresponding to the phonetic character "开" can be the character vector corresponding to the phonetic character "打" output by the feature decoder.

[0073] In this embodiment, the adaptive decoder can implement step S206 through the following sub-steps.

[0074] Sub-step S2061: For each voice character, a first self-attention calculation is performed based on the semantic feature code obtained by performing semantic feature extraction on the audio feature data, the accent feature vector, and the self-attention calculation result of the character vector of the previous voice character of the voice character to be recognized.

[0075] The result of the first self-attention calculation can be recorded as query1, and its calculation process can be expressed as query1=MAH(H asr ,F(y n-1 ), H ad ). Wherein, MAH() represents the first self-attention calculation, which can be a multi-head self-attention calculation. asr is the semantic feature encoding vector. n-1 is the character vector of the previous voice character, F() can be n-1 Perform self-attention calculation, F(y n-1 ) is the result of self-attention calculation on the character vector of the previous voice character. ad The attribute feature vector may include at least an accent feature vector and may also include a speaker identifier, etc., which is not limited to this. In this example, the accent feature vector is mainly illustrated.

[0076] During the first self-attention calculation, the character vector output by the feature decoder is reused. Therefore, the adaptive decoder can synchronize with the character window of the feature decoder by combining the character vector and the accent feature vector output by the adaptive encoder. That is, the speech character corresponding to the adaptive vector determined by the adaptive decoder is synchronized with the speech character for which the feature decoder needs to determine the character vector, thereby outputting the adaptive vector at the speech character level, which can meet the recognition requirements of the feature decoder and ensure recognition accuracy.

[0077] Sub-step S2062: performing a second self-attention calculation based on the calculation result of the first self-attention calculation.

[0078] The result of the second self-attention calculation can be denoted as query2, and the second self-attention calculation can be denoted as:

[0079] query2=MAH(query1, query1, query1). Wherein, MAH() represents the second self-attention calculation, which can be a multi-head self-attention calculation. Through the second self-attention calculation, the deep features in the calculation result of the first self-attention calculation can be further extracted.

[0080] Sub-step S2063: performing a third self-attention calculation based on the result of the second self-attention calculation and the accent feature code corresponding to the accent feature vector.

[0081] The result of the third self-attention calculation can be denoted as query3, and the third self-attention calculation can be denoted as: query3=MHA(query2, H ad , H ad ). Wherein, MAH() represents the third self-attention calculation, which can be a multi-head self-attention calculation. Through the third self-attention calculation, the context information corresponding to the previous speech character can be further extracted from the accent feature vector by using query2, that is, the adaptive vector corresponding to the current speech character.

[0082] It should be noted that the first, second and third self-attention calculations can adopt the same calculation method, or different calculation methods, and the specific implementation form of the self-attention calculation is not limited by the embodiments of the present application.

[0083] Sub-step S2064: predicting and generating the adaptive vector of the speech character to be recognized according to the result of the third self-attention calculation.

[0084] In one possible way, the result of the third self-attention calculation can be used as the adaptive vector. Alternatively, the adaptive vector can be determined by performing dimension transformation and other processing on the result of the third self-attention calculation.

[0085] Step S208: obtaining the attribute feature corresponding to the adaptive vector from the attribute feature library in which the speech attribute features are stored.

[0086] In this embodiment, step S208 can be implemented as follows: inputting the adaptive vector into an attribute feature library storing speech attribute features for matching, obtaining matching attribute features, and outputting a label synchronization vector (label synchronization embedding) corresponding to the matching attribute features.

[0087] For example, based on the adaptive vector, attribute features corresponding to the adaptive vector are matched from an attribute feature library storing speech attribute features through multi-head self-attention calculation. By performing multi-head self-attention calculation on the adaptive vector and the attribute features in the attribute feature library, the degree of match between the adaptive vector and each attribute feature can be determined, thereby determining the attribute feature corresponding to the adaptive vector.

[0088] Alternatively, the attribute feature that matches the adaptive vector may be determined by calculating a cosine value between the adaptive vector and the attribute feature.

[0089] In one feasible approach, the matching attribute features can be directly used as label synchronization vectors; in another feasible approach, the matching attribute features can be used as label synchronization vectors after being aligned with character vectors, such as dimension alignment.

[0090] Step S210: obtaining a recognition result of each phonetic character according to the attribute feature and the initial vector.

[0091] In one feasible approach, for each spoken character, the attribute features corresponding to the current spoken character are input into a feature decoder. The feature decoder then fuses the attribute features of the current spoken character with the initial vector to perform adaptive recognition of the speech to be recognized, thereby outputting the character vector corresponding to the current spoken character. The character vector of the current spoken character can be used to determine its corresponding text and, at the same time, serve as the initial vector for the next spoken character to be recognized. This achieves autoregressive speech recognition and improves recognition accuracy.

[0092] Step S212: according to the recognition result of each voice character and the speaker identifier corresponding to the voice to be recognized, the speaker identifier and the overall recognition result of the voice to be recognized are correlated and output.

[0093] After obtaining the recognition results for each phonetic character in the speech to be recognized, in order to easily determine the speaker, the speaker identifier and the recognition results of each phonetic character can be associated and output in word order. The overall recognition result can be: "Speaker 1: The weather is really nice today."

[0094] Through this embodiment, matching attribute features are retrieved from the attribute feature library based on the adaptive vector output by the adaptive decoder, achieving unsupervised extraction of attribute features of the speech to be recognized. These attribute features can characterize the accent features, pronunciation features, and so on of the speech to be recognized. This allows accurate, voice-character-level accent features to be extracted without the need for a classifier, resolving the problem of inaccurate classification of the accent of the speech to be recognized, which leads to reduced adaptability of recognition. By extracting adaptive vectors of the voice characters of the speech to be recognized, different accents and different speakers can be adaptively recognized, thereby improving the accuracy of speech recognition. Since this embodiment does not require pre-training of the classifier, it can be "plug and play", making it more immediate to use.

[0095] Example 3

[0096] Reference Figure 3A , shows a schematic flow chart of the steps of the speech recognition method of Example 3 of the present application.

[0097] In this embodiment, the method includes the following steps:

[0098] Step S302: receiving a speech to be recognized, and sending the speech to be recognized to the cloud by calling an interface, so that the cloud performs speech recognition on the speech to be recognized.

[0099] The cloud can perform speech recognition on the speech to be recognized according to the speech recognition method as described in the above embodiments.

[0100] In one feasible approach, the method of this embodiment can be executed by a smart device, such as a smart speaker, smart TV, smart router, or smart conferencing device. The smart device receives the speech to be recognized and sends it to the cloud by calling a preset interface. The cloud performs speech recognition on the speech to be recognized, thereby identifying the text corresponding to the speech to be recognized as the speech recognition result. The cloud can then send the speech recognition result to the smart device.

[0101] Step S304: Receive the speech recognition result returned by the cloud.

[0102] The smart device receives the voice recognition results returned from the cloud and records or stores them. In addition, the voice recognition results can be displayed for easy viewing.

[0103] Figure 3B A schematic diagram of a speech recognition method scenario is shown. The smart device includes a microphone and a processor, which are connected to the cloud and the microphone, respectively. The microphone captures the speech to be recognized and sends it to the processor. The processor receives the speech to be recognized and sends it to the cloud. The cloud recognizes the speech to be recognized using the aforementioned speech recognition method, thereby obtaining corresponding text as the speech recognition result. The cloud sends the speech recognition result to the processor. The processor can store the result or directly display it on a connected display device for easy viewing.

[0104] To improve prediction accuracy and address issues like homophones and variant characters, the cloud in this embodiment is equipped with an autoregressive speech recognition model. This model's feature decoder employs autoregressive prediction. Specifically, during prediction, the feature decoder predicts the character vector of the current spoken character based on the results of semantic feature extraction, the extracted accent feature vector, and the character vector predicted for the previous spoken character. This approach, while taking into account the accent of the speech to be recognized, fully incorporates contextual information from the speech to be recognized and the character vector of the predicted speech character, helping to improve accuracy.

[0105] Example 4

[0106] Reference Figure 4A , shows a schematic flow chart of the steps of the speech recognition method of embodiment 4 of the present application.

[0107] The method comprises the following steps:

[0108] Step S402: collecting conference voice in real time, and sending the conference voice to the cloud by calling an interface, so that the cloud performs voice recognition on the conference voice.

[0109] Specifically, the cloud can perform speech recognition on the conference speech according to the speech recognition method as described above.

[0110] For example, a device equipped with a sound collection device collects conference audio and sends the conference audio to the cloud through a preset interface. The conference audio is used as the audio to be recognized, and the cloud is recognized using the method of the first or second embodiment described above, and the text corresponding to the conference audio is obtained as the audio recognition result.

[0111] The cloud can return the speech recognition results to the device that collects the conference audio, or send them to other third-party devices.

[0112] Step S404: Receive the voice recognition result returned by the cloud, and display the meeting minutes based on the voice recognition result.

[0113] The display of the conference record based on the speech recognition result includes displaying the information of the current conference speaker and the speech recognition result corresponding to the speech of the current conference speaker in real time through a display interface.

[0114] The information of the current conference speaker may include the identifier of the current conference speaker (such as nickname, avatar, etc.). If the current conference speaker has turned on the video option, the information may also include the video image of the current conference speaker, but is not limited thereto. Figure 4B A schematic diagram of an interface showing the current conference speakers and their corresponding speech recognition results is shown.

[0115] Alternatively, the display of the meeting minutes based on the speech recognition result includes generating a meeting minute based on the speech recognition result and a preset meeting minute template and displaying the meeting minute through a display interface. Figure 4C A schematic diagram showing a display of meeting minutes is shown.

[0116] This method can easily collect conference voice and perform voice recognition on the conference voice through the cloud, so as to obtain the text corresponding to the conference voice as the voice recognition result, and then display the text corresponding to the conference voice in the display interface for easy viewing.

[0117] Example 5

[0118] Figure 5 The structural block diagram of the speech recognition device of the fourth embodiment of the present application is shown.

[0119] The device includes:

[0120] An extraction module 502 is configured to extract semantic features and attribute features from the audio feature data of the speech to be recognized, wherein the attribute feature extraction includes at least accent feature extraction;

[0121] A first acquisition module 504 is configured to obtain character information of the speech to be recognized based on the result of the semantic feature extraction;

[0122] A generating module 506 is configured to generate an adaptive vector corresponding to each voice character based on the accent feature vector of the voice to be recognized obtained by extracting the attribute features and the initial vector matched with each voice character determined based on the character information;

[0123] A second acquisition module 508 is configured to acquire an attribute feature corresponding to the adaptive vector from an attribute feature library storing speech attribute features;

[0124] The third acquisition module 510 is configured to obtain a recognition result of each phonetic character according to the attribute feature and the initial vector.

[0125] Optionally, the second obtaining module 508 is configured to store the adaptive vector into an attribute feature library storing voice attribute features for matching, obtain a matched attribute feature, and output a label synchronization vector corresponding to the matched attribute feature.

[0126] Optionally, the second obtaining module 508 is configured to match, based on the adaptive vector, an attribute feature corresponding to the adaptive vector from an attribute feature library storing voice attribute features through multi-head attention calculation.

[0127] Optionally, the generating module 506 is configured to, for each voice character, perform first self-attention calculation based on semantic feature encoding obtained by performing semantic feature extraction on the audio feature data, the accent feature vector, and a self-attention calculation result of a character vector of a previous voice character of the voice character to be recognized; perform second self-attention calculation based on a calculation result of the first self-attention calculation; perform third self-attention calculation based on a result of the second self-attention calculation and accent feature encoding corresponding to the accent feature vector; and predict and generate an adaptive vector of the voice character to be recognized according to a result of the third self-attention calculation.

[0128] Optionally, the device is implemented as an autoregressive speech recognition model; wherein the autoregressive speech recognition model comprises a semantic extraction part configured to perform semantic feature extraction on audio feature data of the voice to be recognized, an adaptive part configured to generate an adaptive vector carrying an accent feature of the voice to be recognized based on the voice to be recognized and obtain an attribute feature corresponding to the adaptive vector, and a recognition output part configured to perform recognition output of the voice to be recognized according to the attribute feature.

[0129] Optionally, the semantic extraction part includes: a feature encoder for extracting semantic features from the audio feature data of the speech to be recognized; the adaptive part includes: an adaptive encoder, an adaptive decoder, a self-attention layer and a label synchronization layer, wherein the adaptive encoder is used to extract attribute features from the audio feature data of the speech to be recognized and output a corresponding encoding vector; the adaptive decoder is used to decode the encoding vector output by the adaptive encoder and receive the initial vector output by the feature decoder in the recognition output part, and generate an adaptive vector corresponding to each speech character based on the decoding result and the initial vector; the self-attention layer is used to match the attribute features corresponding to the adaptive vector output by the adaptive decoder from an attribute feature library storing speech attribute features through self-attention calculation; the label synchronization layer is used to generate a label synchronization vector based on the matched attribute features and input it to the feature decoder of the recognition output part; the recognition output part includes: a feature decoder, and the feature decoder is used to output the recognition result of each speech character based on the initial vector and the label synchronization vector corresponding to each speech character.

[0130] Optionally, the attribute feature extraction further includes extracting a speaker identifier corresponding to the speech to be recognized. Accordingly, the result obtained by the attribute feature extraction further includes a speaker identifier.

[0131] Optionally, the apparatus further comprises: an output module 512 for outputting, based on the recognition result of each voice character and the speaker identifier corresponding to the voice to be recognized, the associated speaker identifier and the overall recognition result of the voice to be recognized.

[0132] The speech recognition device of this embodiment is used to implement the corresponding speech recognition methods in the aforementioned multiple method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here. In addition, the functional implementation of each module in the speech recognition device of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments, which will not be described in detail here.

[0133] Example 6

[0134] Figure 6 The structural block diagram of the speech recognition device of Example 6 of the present application is shown.

[0135] The device includes:

[0136] The first receiving module 602 is configured to receive a speech to be recognized and send the speech to be recognized to the cloud by calling an interface, so that the cloud performs speech recognition on the speech to be recognized;

[0137] The second receiving module 604 receives the speech recognition result returned by the cloud.

[0138] The speech recognition device of this embodiment is used to implement the corresponding speech recognition methods in the aforementioned multiple method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here. In addition, the functional implementation of each module in the speech recognition device of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments, which will not be described in detail here.

[0139] Example 7

[0140] Figure 7 The structural block diagram of the speech recognition device of the seventh embodiment of the present application is shown.

[0141] The device comprises:

[0142] The collection module 702 is used to collect conference voice in real time and send the conference voice to the cloud by calling an interface so that the cloud can perform voice recognition on the conference voice;

[0143] The third receiving module 704 is used to receive the speech recognition result returned by the cloud and display the meeting record based on the speech recognition result.

[0144] Optionally, when the third receiving module 704 displays the meeting minutes based on the voice recognition results, it displays the information of the current meeting speaker and the voice recognition results corresponding to the speech of the current meeting speaker in real time through the display interface; or, based on the voice recognition results and the preset meeting minutes template, it generates a meeting minute and displays the meeting minute through the display interface.

[0145] The speech recognition device of this embodiment is used to implement the corresponding speech recognition methods in the aforementioned multiple method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here. In addition, the functional implementation of each module in the speech recognition device of this embodiment can refer to the description of the corresponding parts in the aforementioned method embodiments, which will not be described in detail here.

[0146] Example 8

[0147] Reference Figure 8 , shows a structural diagram of an electronic device according to embodiment eight of the present application. The specific embodiments of the present application do not limit the specific implementation of the electronic device.

[0148] like Figure 8 As shown, the electronic device may include: a processor (processor) 802 , a communication interface (Communications Interface) 804 , a memory (memory) 806 , and a communication bus 808 .

[0149] in:

[0150] The processor 802 , the communication interface 804 , and the memory 806 communicate with each other via a communication bus 808 .

[0151] The communication interface 804 is used to communicate with other electronic devices or servers.

[0152] The processor 802 is configured to execute the program 810 , and specifically may execute the relevant steps in the above method embodiment.

[0153] Specifically, the program 810 may include program codes, which include computer operation instructions.

[0154] The processor 802 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs, or processors of different types, such as one or more CPUs and one or more ASICs.

[0155] The memory 806 is used to store the program 810. The memory 806 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0156] The program 810 may be specifically configured to enable the processor 802 to perform operations corresponding to any one of the methods described in the aforementioned multiple method embodiments.

[0157] The specific implementation of each step in program 810 can refer to the corresponding description of the corresponding steps and units in the above-mentioned method embodiment, and will not be repeated here. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working process of the above-mentioned devices and modules can refer to the corresponding process description in the above-mentioned method embodiment, and will not be repeated here.

[0158] An embodiment of the present application also provides a computer program product, including computer instructions, which instruct a computing device to execute operations corresponding to any one of the above-mentioned multiple method embodiments.

[0159] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0160] The above-mentioned method according to the embodiment of the present application can be implemented in hardware, firmware, or can be implemented as software or computer code that can be stored in a recording medium (such as CD ROM, RAM, floppy disk, hard disk or magneto-optical disk), or can be implemented as computer code that is originally stored in a remote recording medium or a non-temporary machine-readable medium downloaded via a network and will be stored in a local recording medium, so that the method described herein can be stored in such software processing on a recording medium using a general-purpose computer, a special-purpose processor or programmable or special-purpose hardware (such as ASIC or FPGA). It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code, and when the software or computer code is accessed and executed by a computer, a processor or hardware, the method described herein is implemented. In addition, when a general-purpose computer accesses the code for implementing the method shown here, the execution of the code converts the general-purpose computer into a special-purpose computer for executing the method shown here.

[0161] Those skilled in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of this application.

[0162] The above implementation methods are only used to illustrate the embodiments of the present application, and are not intended to limit the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present application, and the scope of patent protection of the embodiments of the present application should be defined by the claims.< / en> < / cn> < / en> < / cn> < / eos> < / sos> < / en> < / cn>

Claims

1. A speech recognition method, comprising: Performing semantic feature extraction and attribute feature extraction on the audio feature data of the speech to be recognized, wherein the attribute feature extraction includes at least accent feature extraction; Obtaining character information of the speech to be recognized based on the result of the semantic feature extraction; Generating an adaptive vector corresponding to each voice character based on the accent feature vector of the voice to be recognized obtained by extracting the attribute features and an initial vector matched with each voice character determined based on the character information, wherein the initial vector is used to represent the character vector of the recognized voice character; Acquiring an attribute feature corresponding to the adaptive vector from an attribute feature library storing speech attribute features; Obtaining a recognition result for each phonetic character according to the attribute features corresponding to the adaptive vector and the initial vector; Among them, the accent feature vector of the speech to be recognized obtained according to the attribute feature extraction and the initial vector matched with each speech character determined according to the character information are used to generate an adaptive vector corresponding to each speech character, including: for each speech character, a first self-attention calculation is performed based on the semantic feature code obtained by semantic feature extraction of the audio feature data, the accent feature vector, and the self-attention calculation result of the character vector of the previous speech character of the speech character to be recognized; a second self-attention calculation is performed based on the result of the first self-attention calculation; a third self-attention calculation is performed based on the result of the second self-attention calculation and the accent feature code corresponding to the accent feature vector; and according to the result of the third self-attention calculation, the adaptive vector of the speech character to be recognized is predicted and generated.

2. The method according to claim 1, wherein The step of acquiring the attribute feature corresponding to the adaptive vector from an attribute feature library storing speech attribute features includes: The adaptive vector is input into an attribute feature library storing speech attribute features for matching, a matched attribute feature is obtained, and a label synchronization vector corresponding to the matched attribute feature is output.

3. The method according to claim 2, wherein: The step of inputting the adaptive vector into an attribute feature library storing speech attribute features for matching to obtain matching attribute features includes: Based on the adaptive vector, attribute features corresponding to the adaptive vector are matched from an attribute feature library storing speech attribute features through multi-head attention calculation.

4. The method according to any one of claims 1 to 3, wherein: The speech recognition method is performed by an autoregressive speech recognition model; Among them, the autoregressive speech recognition model includes: a semantic extraction part for extracting semantic features from the audio feature data of the speech to be recognized, an adaptive part for generating an adaptive vector carrying the accent features of the speech to be recognized based on the speech to be recognized and obtaining attribute features corresponding to the adaptive vector, and a recognition output part for performing recognition output of the speech to be recognized based on the attribute features corresponding to the adaptive vector.

5. The method according to claim 4, wherein The semantic extraction part includes: a feature encoder for extracting semantic features from the audio feature data of the speech to be recognized; The adaptive part includes: an adaptive encoder, an adaptive decoder, a self-attention layer and a label synchronization layer, wherein the adaptive encoder is used to extract attribute features from the audio feature data of the speech to be recognized and output a corresponding encoding vector; the adaptive decoder is used to decode the encoding vector output by the adaptive encoder and receive the initial vector output by the feature decoder in the recognition output part, and generate an adaptive vector corresponding to each speech character according to the decoding result and the initial vector; the self-attention layer is used to match the attribute features corresponding to the adaptive vector from the attribute feature library storing speech attribute features through self-attention calculation on the adaptive vector output by the adaptive decoder; the label synchronization layer is used to generate a label synchronization vector according to the matched attribute features and input it into the feature decoder of the recognition output part; The recognition output part includes: a feature decoder, which is used to output the recognition result of each voice character according to the initial vector and the label synchronization vector corresponding to each voice character.

6. The method according to claim 1, wherein The attribute feature extraction further includes extracting a speaker identifier corresponding to the speech to be recognized. Accordingly, the result obtained by the attribute feature extraction also includes a speaker identifier.

7. The method according to claim 6, wherein: The method further comprises: According to the recognition result of each voice character and the speaker identifier corresponding to the voice to be recognized, the speaker identifier and the overall recognition result of the voice to be recognized are correlated and output.

8. A speech recognition method, comprising: Receive a speech to be recognized, and send the speech to be recognized to the cloud by calling an interface, so that the cloud performs speech recognition on the speech to be recognized according to the method according to any one of claims 1 to 7; Receive the speech recognition result returned by the cloud.

9. A speech recognition method, comprising: Collecting conference voice in real time, and sending the conference voice to the cloud by calling an interface, so that the cloud performs voice recognition on the conference voice according to the voice recognition method according to any one of claims 1 to 7; Receive the voice recognition result returned by the cloud, and display the meeting minutes based on the voice recognition result.

10. The method according to claim 9, wherein: The displaying of the meeting minutes based on the speech recognition result includes: The information of the current conference speaker and the speech recognition results corresponding to the current conference speaker's speech are displayed in real time through the display interface; or, A meeting record is generated according to the speech recognition result and a preset meeting record template, and the meeting record is displayed through a display interface.

11. A speech recognition method, comprising: The semantic extraction part of the autoregressive speech recognition model is used to extract semantic features from the audio feature data of the speech to be recognized; generating an adaptive vector carrying the accent feature of the speech to be recognized based on the speech to be recognized by the adaptive part of the autoregressive speech recognition model and obtaining attribute features corresponding to the adaptive vector; Recognition output of the speech to be recognized is performed according to the attribute features corresponding to the adaptive vector by the recognition output part of the autoregressive speech recognition model; Among them, generating an adaptive vector carrying the accent features of the speech to be recognized based on the speech to be recognized includes: for each speech character, performing a first self-attention calculation based on the semantic feature code obtained by semantic feature extraction of the audio feature data, the accent feature vector of the speech to be recognized obtained by attribute feature extraction, and the self-attention calculation result of the character vector of the previous speech character of the speech character to be recognized; performing a second self-attention calculation based on the result of the first self-attention calculation; performing a third self-attention calculation based on the result of the second self-attention calculation and the accent feature code corresponding to the accent feature vector; and predicting and generating the adaptive vector of the speech character to be recognized based on the result of the third self-attention calculation.

12. The method according to claim 11, wherein The method of extracting semantic features from the audio feature data of the speech to be recognized by the semantic extraction part of the autoregressive speech recognition model comprises: extracting semantic features from the audio feature data of the speech to be recognized by the feature encoder of the semantic extraction part of the autoregressive speech recognition model; The adaptive part of the autoregressive speech recognition model generates an adaptive vector carrying the accent features of the speech to be recognized based on the speech to be recognized and obtains the attribute features corresponding to the adaptive vector, including: extracting the attribute features of the audio feature data of the speech to be recognized by the adaptive encoder of the adaptive part to obtain the output coding vector; decoding the coding vector output by the adaptive encoder by the adaptive decoder of the adaptive part and receiving the initial vector output by the feature decoder in the recognition output part of the autoregressive speech recognition model, and generating an adaptive vector corresponding to each speech character according to the decoding result and the initial vector; matching the attribute features corresponding to the adaptive vector from the attribute feature library storing speech attribute features through the self-attention layer of the adaptive part by self-attention calculation on the adaptive vector output by the adaptive decoder; generating a label synchronization vector according to the matched attribute features by the label synchronization layer of the adaptive part and inputting it into the feature decoder of the recognition output part; The recognition output part of the autoregressive speech recognition model performs recognition output of the speech to be recognized according to the attribute features corresponding to the adaptive vector, including: outputting the recognition result of each speech character according to the initial vector and the label synchronization vector corresponding to each speech character through the feature decoder of the recognition output part.

13. A computer storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the speech recognition method according to any one of claims 1 to 12 is implemented.

Images