Speech recognition method and device, storage medium and electronic equipment

Abstract

The invention provides a voice recognition method and device, a storage medium and electronic equipment, and the method comprises the steps: calling a target pre-training model, carrying out the feature extraction of target voice data, and obtaining the voice feature of each target voice frame in the target voice data; based on the voice feature of each target voice frame, performing clustering processing on the target voice data to obtain a plurality of clustered voice features so as to determine at least one text recognition result; when the target voice data meets the streaming audio data recognition condition, determining audio data representation features under the target voice data based on the plurality of clustering voice features and the at least one text recognition result; and calling a target attention model, and determining a decoding feature of each text recognition result in the at least one text recognition result based on the audio data representation features under the target voice data, thereby determining a target voice recognition result of the target voice data. According to the embodiment of the invention, voice recognition can be quickly carried out on the target voice data.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a speech recognition method, device, storage medium and electronic equipment. Background Art

[0002] At present, speech recognition technology has been widely used, such as in smart phones, smart speakers, smart TVs and other devices; among them, speech recognition is a technology that converts speech into text. However, related technologies usually need to receive complete speech content before starting speech recognition, and cannot achieve real-time, streaming speech recognition, resulting in significant time delay. Based on this, how to quickly perform speech recognition to improve speech recognition efficiency has not yet had a good solution. Summary of the invention

[0003] In view of this, the embodiments of the present invention provide a speech recognition method, device, storage medium and electronic device to solve the problems such as low efficiency of speech recognition caused by related technologies; that is, the embodiments of the present invention can quickly perform speech recognition on the target voice data when it is detected that the target voice data meets the streaming audio data recognition conditions, without having to wait for the complete voice data to be acquired before starting speech recognition, so as to effectively improve the speech recognition efficiency and effectively reduce the time delay of speech recognition, thereby improving the user experience.

[0004] According to one aspect of an embodiment of the present invention, a speech recognition method is provided, the method comprising:

[0005] Acquire target voice data, wherein the target voice data includes voice data between a target start time point and a current time point;

[0006] Calling a target pre-trained model to perform feature extraction on the target speech data to obtain speech features of each target speech frame in the target speech data;

[0007] Based on the speech features of each target speech frame, clustering the target speech data to obtain a plurality of clustered speech features; and based on the plurality of clustered speech features, determining at least one text recognition result;

[0008] When the target voice data satisfies the streaming audio data recognition condition, determining the audio data representation feature under the target voice data based on the multiple clustered voice features and the at least one text recognition result;

[0009] Call the target attention model to determine the decoding features of each text recognition result in the at least one text recognition result based on the audio data representation features under the target speech data; and determine the target speech recognition result of the target speech data based on the decoding features of each text recognition result.

[0010] According to another aspect of an embodiment of the present invention, a speech recognition device is provided, the device comprising:

[0011] An acquisition unit, configured to acquire target voice data, wherein the target voice data includes voice data between a target start time point and a current time point;

[0012] A processing unit, used to call a target pre-training model, perform feature extraction on the target speech data, and obtain speech features of each target speech frame in the target speech data;

[0013] The processing unit is further configured to perform clustering processing on the target speech data based on the speech features of each target speech frame to obtain a plurality of clustered speech features; and determine at least one text recognition result based on the plurality of clustered speech features;

[0014] The processing unit is further configured to determine, when the target voice data satisfies a streaming audio data recognition condition, an audio data representation feature under the target voice data based on the multiple clustered voice features and the at least one text recognition result;

[0015] The processing unit is also used to call the target attention model to determine the decoding features of each text recognition result in the at least one text recognition result based on the audio data representation features under the target speech data; and determine the target speech recognition result of the target speech data based on the decoding features of each text recognition result.

[0016] According to another aspect of an embodiment of the present invention, an electronic device is provided, comprising a processor and a memory storing a program, wherein the program comprises instructions, which, when executed by the processor, enable the processor to perform the above-mentioned method.

[0017] According to another aspect of an embodiment of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above-mentioned method.

[0018] After acquiring the target voice data, the embodiment of the present invention can call the target pre-training model to extract the features of the target voice data to obtain the voice features of each target voice frame in the target voice data; then, based on the voice features of each target voice frame, the target voice data can be clustered to obtain multiple clustered voice features; and based on the multiple clustered voice features, at least one text recognition result can be determined; and based on the multiple clustered voice features, at least one text recognition result can be determined. Correspondingly, when the target voice data meets the streaming audio data recognition condition, the audio data representation feature under the target voice data is determined based on the multiple clustered voice features and at least one text recognition result; the target attention model is called to determine the decoding feature of each text recognition result in at least one text recognition result based on the audio data representation feature under the target voice data; and based on the decoding feature of each text recognition result, the target voice recognition result of the target voice data is determined. It can be seen that the embodiment of the present invention can quickly perform voice recognition on the target voice data when it is detected that the target voice data meets the streaming audio data recognition condition, without waiting for the complete voice data to be acquired before starting voice recognition, so as to effectively improve the voice recognition efficiency, and effectively reduce the time delay of voice recognition, thereby improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Further details, features and advantages of the invention are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0020] Figure 1 A flow chart of a speech recognition method according to an exemplary embodiment of the present invention is shown;

[0021] Figure 2 A schematic flow chart of another speech recognition method according to an exemplary embodiment of the present invention is shown;

[0022] Figure 3 A schematic flow chart of another speech recognition method according to an exemplary embodiment of the present invention is shown;

[0023] Figure 4 A schematic block diagram of a speech recognition device according to an exemplary embodiment of the present invention is shown;

[0024] Figure 5 A block diagram of an exemplary electronic device that can be used to implement an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0025] Embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present invention. It should be understood that the drawings and embodiments of the present invention are only for exemplary purposes and are not intended to limit the scope of protection of the present invention.

[0026] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0027] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc. mentioned in the present invention are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0028] It should be noted that the modifications of "one" and "plurality" mentioned in the present invention are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0029] The names of the messages or information exchanged between multiple devices in the embodiments of the present invention are only used for illustrative purposes, and are not used to limit the scope of these messages or information.

[0030] It should be noted that the execution subject of the speech recognition method provided in the embodiment of the present invention may be one or more electronic devices, which is not limited by the present invention; wherein, the electronic device may be a terminal (i.e., a client) or a server, then when the execution subject includes multiple electronic devices, and the multiple electronic devices include at least one terminal and at least one server, the speech recognition method provided in the embodiment of the present invention may be jointly executed by the terminal and the server. Accordingly, the terminal mentioned here may include but is not limited to: smart phones, tablet computers, laptop computers, desktop computers, smart watches, intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, etc. The server mentioned here may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing (cloud computing), cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and basic cloud computing services such as big data and artificial intelligence platforms, etc.

[0031] Based on the above description, an embodiment of the present invention proposes a speech recognition method, which can be executed by the electronic device (terminal or server) mentioned above; or, the speech recognition method can be executed by the terminal and the server together. For the sake of convenience, the following description will be given by taking the electronic device executing the speech recognition method as an example; Figure 1 As shown, the speech recognition method may include the following steps S101-S105:

[0032] S101, acquiring target voice data, where the target voice data includes voice data between a target start time point and a current time point.

[0033] In the embodiment of the present invention, the current time point may be a time point for acquiring target voice data.

[0034] Optionally, the electronic device may trigger the acquisition of target voice data every preset waiting time when the audio input operation is detected; that is, the target voice data may be acquired every preset waiting time according to the target starting time point and the current time point, that is, the voice data between the target starting time point and the current time point may be used as the target voice data to achieve the acquisition of the target voice data, and the end time of the target voice data may be the current time point of acquiring the target voice data; that is, the target voice data may be updated every preset waiting time. Wherein, the target starting time point may be any of the following: the audio starting time of the audio data indicated by the audio input operation, the end time of the previous streaming audio data, the next time point of the end time of the previous streaming audio data (such as when the end time of the previous streaming audio data is the 30th second of the 2nd minute, the next time point of the end time of the previous streaming audio data may be the 31st second of the 2nd minute, etc.); when the target voice data meets the streaming audio data condition, the target starting time point is updated to obtain the starting time point of the next target voice data, that is, the next target voice data may be acquired through the current target starting time point (i.e., the updated target starting time point). In the embodiment of the present invention, a stream audio data may be a stream data, and may also be referred to as stream voice data, audio data stream or segmented audio data, etc. Optionally, the voice recognition mentioned in the embodiment of the present invention may also be referred to as stream voice recognition. Optionally, the target start time point may initially be the audio start time of the audio data indicated by the audio input operation.

[0035] Optionally, the preset waiting time can be set according to experience or according to actual needs, and the embodiment of the present invention does not limit this. Optionally, the speech recognition method mentioned in the embodiment of the present invention can be applied to scenarios such as intelligent customer service, robot letter review, and speech-to-text conversion, and the embodiment of the present invention does not limit this; Optionally, the electronic device can automatically recognize the speech data into text and pass the text to subsequent tasks.

[0036] Based on this, the user can perform an audio input operation to input voice data through the audio input operation, so that the electronic device detects the audio input operation. Exemplarily, any display interface of the electronic device may include a voice input button, then the user can perform a long press operation or a click operation on the voice input button to input voice data, thereby implementing the audio input operation, such as inputting voice data while pressing the voice input button to implement the audio input operation; or, the voice input button can be clicked to make the audio input state be the inputting state, thereby inputting voice data to implement the audio input operation, and so on; the embodiment of the present invention is not limited to this.

[0037] In other embodiments, when receiving voice data sent by other devices, the electronic device may also obtain target voice data from the received voice data at preset waiting time intervals; or, it may read the target voice data from the voice data to be processed in its own storage space, and so on; the present invention is not limited to this.

[0038] S102, calling the target pre-training model, performing feature extraction on the target speech data, and obtaining speech features of each target speech frame in the target speech data.

[0039] Optionally, the target pre-trained model can be a pre-trained model; optionally, a pre-trained model can be a Hubert model (a speech recognition pre-trained model), or a wav2vec model (another speech recognition pre-trained model), etc., which is not limited in the embodiments of the present invention. Optionally, the electronic device can obtain the target pre-trained model (also referred to as the target speech recognition pre-trained model) from its own storage space; or, a pre-trained model download link can be obtained to use the pre-trained model downloaded based on the pre-trained model download link as the target pre-trained model; or, a training speech data set can be used to iteratively train the initial pre-trained model to obtain the target pre-trained model, etc.; the embodiments of the present invention are not limited in this regard.

[0040] S103, clustering the target speech data based on the speech features of each target speech frame to obtain a plurality of clustered speech features; and determining at least one text recognition result based on the plurality of clustered speech features.

[0041] In an embodiment of the present invention, discrete speech features can be obtained by clustering speech features, and speech data can be processed by aligning discrete speech features with text, so that the discrete speech features can be better aligned with the text modality, thereby improving the accuracy of text recognition results and the audio data representation features under the target speech data described below.

[0042] Optionally, the electronic device may adopt a K-means clustering method to perform clustering processing on the target speech data based on the speech features of each target speech frame (i.e., clustering processing is performed on the speech frames in the target speech data, which may also be referred to as clustering processing on the speech features of the speech frames in the target speech data) to achieve the generation of discrete speech features using K-means clustering; that is, K-means clustering may be performed on the target speech data based on the speech features of each target speech frame to obtain multiple clustered speech features, and a clustered speech feature may be determined based on the speech features of each speech frame in at least one speech frame in the target speech data, such as the mean between the speech features of each speech frame in at least one speech frame included in the target speech data, that is, it may be a cluster center under a clustering category.

[0043] Optionally, the electronic device can estimate the number of clusters (i.e., K value) by the elbow method, or by the silhouette coefficient method, etc., which is not limited in the embodiment of the present invention. Based on this, multiple clustered speech features (i.e., K clustered speech features) can be obtained, where K is an integer greater than 1.

[0044] S104: When the target voice data meets the streaming audio data recognition condition, determine the audio data representation feature under the target voice data based on the multiple clustered voice features and at least one text recognition result.

[0045] In an embodiment of the present invention, when the target voice data meets the streaming audio data recognition condition, the target voice data can be used as a streaming audio data. Optionally, when at least one text recognition result includes a target boundary marker, it can be determined that the target voice data meets the streaming audio data recognition condition; or, when it is determined that the target voice data is the last segment of voice data in the voice data to be recognized (that is, the end time of the target voice data is the end time of the voice data to be recognized), it can be determined that the target voice data meets the streaming audio data recognition condition, and so on; the embodiment of the present invention is not limited to this. Optionally, if the target voice data is obtained from the audio data indicated by the audio input operation (that is, the input audio data), the voice data to be recognized can be the audio data indicated by the audio input operation. Optionally, the target boundary marker can be any specified character; optionally, the target boundary marker can be set according to experience or according to actual needs, and the embodiment of the present invention is not limited to this.

[0046] Exemplarily, assuming that the preset waiting time is 3 seconds (s), then when the electronic device detects the audio input operation, it can use the voice data of the first 3 seconds input by the audio input operation as the target voice data; assuming that the text recognition result corresponding to the voice data of the first 3 seconds includes a target boundary mark, then the target voice data at this time can be used as streaming audio data, and taking the target starting time point being updated to the next time point after the end time of the previous streaming audio data as an example, the electronic device can obtain the voice data of the 4th to 6th seconds input by the audio input operation as the target voice data. If the text recognition result corresponding to the voice data of the 4th to 6th seconds does not include the target boundary mark at this time, the voice data of the 4th to 9th seconds can be obtained as the target voice data, until the voice data input by the audio input operation is obtained, and so on. Optionally, when the currently acquired voice data (i.e., the currently acquired target voice data) is the last segment of voice data, that is, when the end time of the current target voice data is the end time of the audio data indicated by the audio input operation, the target voice data at this time can be treated as a streaming audio data; that is, the last segment of voice data in the audio data indicated by the audio input operation can be directly treated as a streaming audio data.

[0047] Optionally, when the target voice data does not meet the streaming audio data recognition condition, the electronic device may not trigger the execution of the above-mentioned determination of the audio data representation feature under the target voice data based on multiple clustered voice features and at least one text recognition result, and further does not trigger the execution of determining the target voice recognition result of the target voice data. Optionally, a text recognition result may be a text, that is, a character.

[0048] Further, when determining the audio data representation features under the target speech data based on multiple clustered speech features and at least one text recognition result, the electronic device may generate a speech embedding feature for each clustered speech feature in the multiple clustered speech features; optionally, each clustered speech feature may be passed through the target speech embedding layer (i.e., each clustered speech feature may be extracted through the target speech embedding layer) to generate a speech embedding feature for the corresponding clustered speech feature, so as to realize the generation of speech embedding features for each clustered speech data, that is, the speech embedding feature for each clustered speech feature may be generated through the target speech embedding layer. Optionally, a speech embedding layer may include but is not limited to at least one fully connected layer, etc., which is not limited in the embodiment of the present invention. Correspondingly, the electronic device can generate a text embedding feature for each text recognition result in at least one text recognition result; optionally, each text recognition result in at least one text recognition result can be respectively mapped through a rule to obtain a text encoding feature (also referred to as a rule mapping feature) of each text recognition result, that is, each text recognition result can be encoded according to the rule mapping to obtain a text encoding feature of each text recognition result, so that the text encoding feature of each text recognition result can be respectively passed through the target text embedding layer to generate a text embedding feature for each text recognition result, that is, the text embedding feature of each text recognition result can be generated through the target text embedding layer. Optionally, a text embedding layer can include but is not limited to at least one fully connected layer, which is not limited in the embodiment of the present invention. Optionally, the rule mapping can be set according to experience or according to actual needs, which is not limited in the embodiment of the present invention. Optionally, both the target speech embedding layer and the target text embedding layer can be set according to experience or actual needs, or can be obtained by training the initial speech embedding layer and the initial text embedding layer (i.e., parameter optimization), which is not limited in the embodiment of the present invention.

[0049] Based on this, the electronic device can concatenate the speech embedding features of each clustered speech feature and the text embedding features of each text recognition result to obtain the audio data representation features under the target speech data. It should be understood that the dimension of a speech embedding feature is the same as the dimension of a text embedding feature; illustratively, assuming that the number of clustered speech features in multiple clustered speech features is K, the number of text recognition results in at least one text recognition result is P, and the dimension of an embedding feature is Q, then the size of the audio data representation features under the target speech data can be (K+P)×Q. It can be seen that the audio data representation features under the target speech data may include the speech embedding features of each clustered speech feature and the text embedding features of each text recognition result.

[0050] S105, calling the target attention model, determining the decoding features of each text recognition result in at least one text recognition result based on the audio data representation features under the target speech data; and determining the target speech recognition result of the target speech data based on the decoding features of each text recognition result.

[0051] Optionally, an attention model can be a self-attention model (such as a transformer model) or a cross-attention model, etc., which is not limited in the embodiment of the present invention. Based on this, the audio data representation features under the target voice data can be passed into the target attention model to output the decoding features of each text recognition result. Optionally, the target attention model can be obtained from the local space, or it can be obtained by model training of the initial attention model, which is not limited in the embodiment of the present invention; it should be understood that after the initial attention model is trained to obtain the target attention model, the target attention model can be stored in the local space to facilitate the subsequent call of the target attention model. Optionally, the initial attention model can be set according to experience or actual needs, or it can be randomly initialized, which is not limited in the embodiment of the present invention. Optionally, the attention model in the embodiment of the present invention can also be called a streaming attention model or a streaming attention large model, and a streaming attention mechanism can be used to make the current text (i.e., at least one text recognition result under the current target voice data) only pay attention to the current time period (i.e., the time period where the current target voice data is located) and / or the voice content before the current time period, so as to maintain consistency during reasoning and achieve streaming voice recognition.

[0052] Optionally, when determining the target speech recognition result of the target speech data based on the decoding features of each text recognition result, the electronic device may determine the decoded text corresponding to each text recognition result from the text feature library based on the decoding features of each text recognition result. The decoded text corresponding to a text (such as a text recognition result) is the text indicated by the matching features of the decoding features of the corresponding text in the text feature library, thereby obtaining the target speech recognition result of the target speech data; optionally, the target speech recognition result may include all decoded texts after deduplication of the decoded text corresponding to each text recognition result. Optionally, the matching feature of a decoding feature in the text feature library may be the text feature in the text feature library that is closest to the decoding feature.

[0053] In an embodiment of the present invention, the audio data indicated by the audio input operation may be the audio data (i.e., voice data) input by the audio input operation, i.e., the complete audio data input by the audio input operation, that is, the audio data input by the user performing an audio input operation. Based on this, the electronic device may divide the audio data indicated by the audio input operation into at least one streaming audio data, so that each streaming audio data in the audio data indicated by the audio input operation (i.e., each streaming audio data in at least one streaming audio data included in the audio data indicated by the audio input operation) may be subjected to voice recognition respectively; that is, each target voice data as streaming audio data may be subjected to voice recognition respectively, so as to obtain a voice recognition result for each streaming audio data.

[0054] Then, accordingly, after obtaining the speech recognition result of each streaming audio data in the audio data indicated by the audio input operation, the speech recognition of the audio data indicated by the audio input operation can be completed. It can be seen that the embodiment of the present invention can start speech recognition of the acquired streaming audio data while the user performs the audio input operation (that is, the audio input operation is not completed), and can output the ideal feasibility without waiting for the complete speech input of the whole sentence.

[0055] After acquiring the target voice data, the embodiment of the present invention can call the target pre-training model to extract the features of the target voice data to obtain the voice features of each target voice frame in the target voice data; then, based on the voice features of each target voice frame, the target voice data can be clustered to obtain multiple clustered voice features; and based on the multiple clustered voice features, at least one text recognition result can be determined; and based on the multiple clustered voice features, at least one text recognition result can be determined. Correspondingly, when the target voice data meets the streaming audio data recognition condition, the audio data representation feature under the target voice data is determined based on the multiple clustered voice features and at least one text recognition result; the target attention model is called to determine the decoding feature of each text recognition result in at least one text recognition result based on the audio data representation feature under the target voice data; and based on the decoding feature of each text recognition result, the target voice recognition result of the target voice data is determined. It can be seen that the embodiment of the present invention can quickly perform voice recognition on the target voice data when it is detected that the target voice data meets the streaming audio data recognition condition, without waiting for the complete voice data to be acquired before starting voice recognition, so as to effectively improve the voice recognition efficiency, and effectively reduce the time delay of voice recognition, thereby improving the user experience.

[0056] Based on the above description, the embodiment of the present invention further proposes a more specific speech recognition method. Accordingly, the speech recognition method can be executed by the electronic device (terminal or server) mentioned above; or, the speech recognition method can be executed by the terminal and the server together. For the sake of convenience, the following description will be given by taking the electronic device executing the speech recognition method as an example; please refer to Figure 2 , the speech recognition method may include the following steps S201-S210:

[0057] S201, obtaining a training speech data set, and obtaining a training label text of each training speech data in the training speech data set, wherein a training label text includes one or more target boundary markers.

[0058] Optionally, a training voice data set and training label text for each training voice data may be stored in the electronic device's own storage space. In this case, the electronic device may obtain the training voice data set from its own storage space, and obtain the training label text for each training voice data; or, may obtain a training data download link to download the training voice data set and the training label text for each training voice data in the training voice data set based on the training data download link, and so on; this is not limited to the embodiments of the present invention.

[0059] S202, for any training speech data in the training speech data set, call the first pre-training model, perform feature extraction on any training speech data, and obtain speech features of each speech frame in any training speech data; and based on the speech features of each speech frame in any training speech data, determine the correspondence between each speech frame in any training speech data and each text in the training label text of any training speech data.

[0060] In an embodiment of the present invention, the target pre-trained model can be determined based on the first pre-trained model; optionally, the target pre-trained model can be the first pre-trained model, or it can be the result of continuing to train the first pre-trained model during the model training of the attention model, and so on; the embodiment of the present invention is not limited to this.

[0061] Optionally, the electronic device may call the initial pre-training model, perform feature extraction on each training voice data respectively, and obtain the training feature extraction result of each training voice data, wherein the training feature extraction result of one training voice data includes the voice features of each voice frame in the corresponding training voice data; and the training feature extraction result of each training voice data is used to calculate the pre-training model loss value, such as the training feature extraction result of each training voice data and the training clustering result of each training voice data can be used to calculate the pre-training model loss value. Optionally, the training clustering result of one training voice data can be the result of clustering the voice frame representation vectors in the corresponding training voice data, and one voice frame representation vector can be used to indicate one voice frame, and feature extraction on one voice data can be feature extraction on the voice frame representation vector of each voice frame in the corresponding voice data.

[0062] Furthermore, the electronic device may optimize the initial pre-trained model in the direction of reducing the loss value of the pre-trained model to obtain a second pre-trained model; and determine the first pre-trained model based on the second pre-trained model. Optionally, the electronic device may continue to call the second pre-trained model, perform feature extraction on each training voice data respectively, obtain the training feature extraction result of each training voice data, and so on, so as to continue to perform model training on the second pre-trained model, thereby realizing iterative training of the pre-trained model until the pre-trained model convergence condition is reached (such as the pre-trained model loss value under the current iteration is less than the pre-trained model loss threshold or the number of iterations reaches the pre-trained model iteration number threshold, etc.), and then the pre-trained model that reaches the pre-trained model convergence condition is used as the first pre-trained model. Optionally, the pre-trained model loss threshold and the pre-trained model iteration number threshold can be set according to experience or according to actual needs, and the embodiments of the present invention are not limited to this.

[0063] In other embodiments, the electronic device may also use voice data other than the training voice data set to perform model training on the pre-trained model, which is not limited in the present invention.

[0064] Optionally, when determining the correspondence between each speech frame in any training speech data and each text in the training label text of any training speech data based on the speech features of each speech frame in any training speech data, the electronic device may perform speech-text alignment (such as through CTC (Connectionist Temporal Classification, a method for solving the problem of different lengths of input and output sequences and inability to align) alignment algorithm, MFA (Montreal-Forced-Aligner, a method for aligning Chinese audio and text), GMM (Gaussian Mixed Model) or HMM (Hidden Mixed Model) on each speech frame in any training speech data and each text in the training label text of any training speech data based on the speech features of each speech frame in any training speech data. MarkovModel, hidden Markov model) etc. are used for speech-text alignment), that is, the speech features of each speech frame in any training speech data and each text in the training label text of any training speech data are aligned with each other to obtain the corresponding relationship (also called mapping relationship) between each speech frame in any training speech data and each text in the training label text of any training speech data; wherein, the corresponding relationship between each speech frame in any training speech data and each text in the training label text of any training speech data can be used to indicate the text corresponding to each speech frame in any training speech data in the training label text of any training speech data (that is, it can be used to indicate the text corresponding to each speech frame in any training speech data), that is, it can be used to indicate the speech frame corresponding to each text in the training label text of any training speech data, and the number of speech frames corresponding to one text can be one or more.

[0065] S203, based on the correspondence between each speech frame in any training speech data and each text in the training label text of any training speech data, divide any training speech data into multiple training streaming audio data, one training streaming audio data corresponds to at least one text in the training label text of any training speech data.

[0066] Among them, a training label text may include multiple texts, and at least one text corresponding to a training streaming audio data may include the text corresponding to each speech frame in the corresponding training streaming audio data. Exemplarily, assuming that a training streaming audio data includes speech frames 1-6, and the texts corresponding to speech frames 1 and 2 in the training streaming audio data are both text 1, and the texts corresponding to speech frames 3-6 are all text 2, then the at least one text corresponding to the training streaming audio data may include text 1 and text 2.

[0067] Optionally, the electronic device may divide the training label text of any training voice data according to the target boundary marker to obtain multiple text division results; and based on the correspondence between each voice frame in any training voice data and each text in the training label text of any training voice data, respectively determine the training streaming audio data corresponding to each text division result in the multiple text division results, so as to realize dividing any training voice data into multiple training streaming audio data, and the multiple training streaming audio data may include the training streaming audio data corresponding to each text division result; wherein the training streaming audio data corresponding to a text division result may include all the voice frames corresponding to each text in the corresponding text division result in any training voice data. Optionally, when any text division result in the multiple text division results is not the last text division result in the training label text of any training voice data, the last text in any text division result may be the target boundary marker, that is, the training label text is divided according to the target boundary marker as the end text of a text division result; when any text division result is the last text division result in the training label text of any training voice data, the last text in any text division result may be the last text in the training label text of any training voice data.

[0068] S204, respectively determine the audio data representation features of each training streaming audio data in multiple training streaming audio data, and call the initial attention model to generate decoding features of each text corresponding to each training streaming audio data based on the audio data representation features of each training streaming audio data, so as to calculate the model loss value under any training speech data based on the decoding features of each text corresponding to each training streaming audio data.

[0069] Optionally, for any training streaming audio data among multiple training streaming audio data, the electronic device may perform clustering processing on any training streaming audio data based on the speech features of each speech frame in any training streaming audio data (that is, clustering processing may be performed on the speech features of the speech frames in any training streaming audio data) to obtain multiple training clustered speech features of any training streaming audio data; then, a speech embedding feature of each training clustered speech feature among the multiple training clustered speech features may be generated, and a text embedding feature of each text in the text segmentation result corresponding to any training streaming audio data may be generated. Optionally, the electronic device can generate a speech embedding feature for each training clustered speech feature through a target speech embedding layer, and generate a text embedding feature for each text in a text segmentation result corresponding to any training streaming audio data through a target text embedding layer; or, can generate a speech embedding feature for each training clustered speech feature through an initial speech embedding layer, and generate a text embedding feature for each text in a text segmentation result corresponding to any training streaming audio data through an initial text embedding layer. In this case, the speech embedding layer (such as the initial speech embedding layer) and the text embedding layer (such as the initial text embedding layer) can be trained simultaneously during the model training of the attention model to obtain the target speech embedding layer and the target text embedding layer.

[0070] Based on this, the electronic device can use the speech embedding features of each training cluster speech feature and the text embedding features of each text in the text segmentation result corresponding to any training streaming audio data to construct the audio data representation features under any training streaming audio data. Accordingly, the speech embedding features of each training cluster speech feature and the text embedding features of each text in the text segmentation result corresponding to any training streaming audio data can be spliced ​​to obtain the audio data representation features under any training streaming audio data.

[0071] In an embodiment of the present invention, each text corresponding to a training streaming audio data includes each text in the text division result corresponding to the corresponding training streaming audio data. Furthermore, when calculating the model loss value under any training speech data based on the decoding features of each text corresponding to each training streaming audio data, for any training streaming audio data among multiple training streaming audio data, the electronic device may calculate the model loss value under any training streaming audio data based on the decoding features of each text corresponding to any training streaming audio data and each text corresponding to any training streaming audio data; optionally, the electronic device may calculate the model loss value under any training streaming audio data based on the decoding features of each text corresponding to any training streaming audio data and each text corresponding to any training streaming audio data through a KL (divergence) error function or a cross entropy loss function, etc.; this is not limited in this embodiment of the present invention. For example, Figure 3As shown, taking the cross entropy loss function as an example, the electronic device can process discrete speech features through clustering, and then obtain the audio data representation features under any training streaming audio data through speech embedding and text embedding, and input the streaming attention model (that is, the current attention model, such as the initial attention model), so as to obtain the decoding features of each text corresponding to any training streaming audio data, and then calculate the model loss value under any training streaming audio data through the cross entropy loss function (which can also be called cross entropy loss at this time), and so on.

[0072] Optionally, an embodiment of the present invention may also adopt a label smoothing strategy to calculate the model loss value under any training streaming audio data by fitting the output of the streaming attention large model (i.e., the attention model) (such as the decoding features of each text corresponding to any training streaming audio data) and the smoothed labels (such as the smoothed each text corresponding to any training streaming audio data) so that the output distribution of the streaming attention large model tends to the output distribution of the actual labels. Based on this, the electronic device can determine the confidence (also referred to as the smoothing value) of each text corresponding to any training streaming audio data, and calculate the model loss value under any training streaming audio data based on the decoding features of each text corresponding to any training streaming audio data, each text corresponding to any training streaming audio data, and the confidence of each text corresponding to any training streaming audio data, so that the confidence of any text is multiplied by the loss term of any text (such as the cross loss value of any text), thereby achieving the smoothing of the text corresponding to any training streaming audio data to improve the accuracy of the model loss value under any training streaming audio data. Optionally, the confidence of each text corresponding to any training streaming audio data can be set according to experience or actual needs, or can be randomly generated within a preset interval, which is not limited in the embodiment of the present invention. Optionally, the preset interval can be set according to experience or according to actual needs, which is not limited in the embodiment of the present invention.

[0073] Based on this, after obtaining the model loss value under each training streaming audio data, the model loss value under each training streaming audio data can be weighted summed (such as mean operation or summation operation, etc.) to obtain the model loss value under any training speech data.

[0074] S205, after obtaining the model loss value under each training speech data, determine the target model loss value, and optimize the model parameters in the initial attention model in the direction of reducing the target model loss value, to obtain the initial attention model after model optimization, and determine the target attention model based on the initial attention model after model optimization.

[0075] In an embodiment of the present invention, the model loss value under each training speech data can be weighted and summed to obtain the target model loss value. Based on this, the electronic device can continue to iteratively train the attention model based on the initial attention model after model optimization until the target convergence condition is reached (such as the number of iterations reaches the attention iteration number threshold or the target model loss value under the current iteration is less than the attention loss threshold, etc.), thereby using the attention model when the target convergence condition is reached as the target attention model. Optionally, the attention iteration number threshold and the attention loss threshold can be set according to experience or according to actual needs, and the embodiment of the present invention is not limited to this.

[0076] Optionally, the electronic device may also optimize the parameters of the initial speech embedding layer, the initial text embedding layer, and at least one of the first pre-trained models in the direction of reducing the target model loss value in each iteration to obtain a target speech embedding layer, a target text embedding layer, and a target pre-trained model, etc. The embodiments of the present invention are not limited to this.

[0077] S206, acquiring target voice data, where the target voice data includes voice data between the target start time point and the current time point.

[0078] S207, calling the target pre-training model, performing feature extraction on the target speech data, and obtaining speech features of each target speech frame in the target speech data.

[0079] S208, clustering the target speech data based on the speech features of each target speech frame to obtain a plurality of clustered speech features; and determining at least one text recognition result based on the plurality of clustered speech features.

[0080] S209: When the target voice data meets the streaming audio data recognition condition, determine the audio data representation feature under the target voice data based on the multiple clustered voice features and at least one text recognition result.

[0081] S210, calling the target attention model, determining the decoding features of each text recognition result in at least one text recognition result based on the audio data representation features under the target speech data; and determining the target speech recognition result of the target speech data based on the decoding features of each text recognition result.

[0082] After obtaining a training speech data set and a training label text for each training speech data in the training speech data set, the embodiment of the present invention can call a first pre-training model for any training speech data in the training speech data set, perform feature extraction on any training speech data, and obtain speech features of each speech frame in any training speech data; and based on the speech features of each speech frame in any training speech data, determine the correspondence between each speech frame in any training speech data and each text in the training label text of any training speech data. Then, based on the correspondence between each speech frame in any training speech data and each text in the training label text of any training speech data, any training speech data can be divided into multiple training streaming audio data, one training streaming audio data corresponds to at least one text in the training label text of any training speech data; and the audio data representation features of each training streaming audio data in the multiple training streaming audio data are determined respectively, and the initial attention model is called to generate the decoding features of each text corresponding to each training streaming audio data based on the audio data representation features of each training streaming audio data, so as to calculate the model loss value of any training speech data based on the decoding features of each text corresponding to each training streaming audio data; after obtaining the model loss value of each training speech data, the target model loss value can be determined, and the model parameters in the initial attention model can be optimized in the direction of reducing the target model loss value to obtain the initial attention model after model optimization, so as to determine the target attention model based on the initial attention model after model optimization. Furthermore, after acquiring the target speech data, the target pre-trained model can be called to extract features of the target speech data to obtain speech features of each target speech frame in the target speech data; then, the target speech data can be clustered based on the speech features of each target speech frame to obtain multiple clustered speech features; and at least one text recognition result can be determined based on the multiple clustered speech features. Based on this, when the target speech data meets the streaming audio data recognition conditions, the audio data representation features under the target speech data can be determined based on the multiple clustered speech features and at least one text recognition result; thereby calling the target attention model, based on the audio data representation features under the target speech data, the decoding features of each text recognition result in at least one text recognition result can be determined; and based on the decoding features of each text recognition result, the target speech recognition result of the target speech data can be determined.It can be seen that the embodiment of the present invention can effectively improve the model performance of the target attention model, thereby improving the accuracy of the target speech recognition result; and the embodiment of the present invention does not need to start speech recognition after receiving the complete speech content, and can realize real-time, streaming speech recognition, which can meet the expectations and needs of users, especially in some occasions that require real-time interaction or rapid response, such as telephone customer service, voice assistant, etc., to improve speech recognition efficiency and improve user experience through streaming output; that is, the embodiment of the present invention can effectively solve the problem that speech recognition in related technologies cannot be streamed, and can output text while receiving audio, that is, while receiving speech input, it can output recognition results in real time, thereby greatly improving the real-time nature of recognition and user experience. Based on this, the embodiment of the present invention can use the streaming attention method to stream-adapt the large model solution, thereby ensuring the real-time reasoning ability of the model, which can effectively solve the problem that the current large model does not have streaming reasoning ability, and the solution mentioned in the embodiment of the present invention can be extended to other different large models.

[0083] Based on the description of the above-mentioned related embodiments of the speech recognition method, the embodiment of the present invention further proposes a speech recognition device, which may be a computer program (including program code) running in an electronic device; Figure 4 As shown, the speech recognition device may include an acquisition unit 401 and a processing unit 402. The speech recognition device may execute Figure 1 or Figure 2 The speech recognition method shown, that is, the speech recognition device can run the above-mentioned units:

[0084] An acquisition unit 401 is used to acquire target voice data, where the target voice data includes voice data between a target start time point and a current time point;

[0085] The processing unit 402 is used to call the target pre-training model, perform feature extraction on the target speech data, and obtain speech features of each target speech frame in the target speech data;

[0086] The processing unit 402 is further configured to perform clustering processing on the target speech data based on the speech features of each target speech frame to obtain a plurality of clustered speech features; and determine at least one text recognition result based on the plurality of clustered speech features;

[0087] The processing unit 402 is further configured to determine, when the target voice data satisfies the streaming audio data recognition condition, an audio data representation feature under the target voice data based on the multiple clustered voice features and the at least one text recognition result;

[0088] The processing unit 402 is also used to call the target attention model to determine the decoding features of each text recognition result in the at least one text recognition result based on the audio data representation features under the target speech data; and determine the target speech recognition result of the target speech data based on the decoding features of each text recognition result.

[0089] In one implementation, when the target voice data meets the streaming audio data recognition condition, the target voice data is treated as streaming audio data, and the acquisition unit 401 may further be used to:

[0090] When an audio input operation is detected, triggering the acquisition of target voice data every preset waiting time;

[0091] The processing unit 402 may also be used to:

[0092] After obtaining the speech recognition result of each stream audio data in the audio data indicated by the audio input operation, completing the speech recognition of the audio data indicated by the audio input operation;

[0093] Among them, the target starting time point is any one of the following: the audio start time of the audio data indicated by the audio input operation, the end time of the previous streaming audio data, and the next time point after the end time of the previous streaming audio data; when the target voice data meets the streaming audio data recognition conditions, the target starting time point is updated to obtain the starting time point of the next target voice data.

[0094] In another implementation, the acquisition unit 401 may also be used to:

[0095] Acquire a training speech data set, and acquire a training label text of each training speech data in the training speech data set, wherein a training label text includes one or more target boundary markers;

[0096] The processing unit 402 may also be used to:

[0097] For any training speech data in the training speech data set, calling the first pre-training model, performing feature extraction on the any training speech data, and obtaining speech features of each speech frame in the any training speech data; and based on the speech features of each speech frame in the any training speech data, determining the correspondence between each speech frame in the any training speech data and each text in the training label text of the any training speech data;

[0098] Based on the correspondence between each speech frame in the any training speech data and each text in the training label text of the any training speech data, dividing the any training speech data into a plurality of training streaming audio data, one training streaming audio data corresponds to at least one text in the training label text of the any training speech data;

[0099] Respectively determine the audio data representation features of each training streaming audio data in the multiple training streaming audio data, and call the initial attention model to generate the decoding features of each text corresponding to each training streaming audio data based on the audio data representation features of each training streaming audio data, so as to calculate the model loss value under any training speech data based on the decoding features of each text corresponding to each training streaming audio data;

[0100] After obtaining the model loss value under each training speech data, the target model loss value is determined, and the model parameters in the initial attention model are optimized in the direction of reducing the target model loss value to obtain the initial attention model after model optimization, so as to determine the target attention model based on the initial attention model after model optimization.

[0101] In another implementation manner, when the processing unit 402 divides the any training speech data into a plurality of training streaming audio data based on the correspondence between each speech frame in the any training speech data and each text in the training label text of the any training speech data, it can be specifically used to:

[0102] According to the target boundary marker, the training label text of any training speech data is divided to obtain multiple text division results;

[0103] Based on the correspondence between each speech frame in any one of the training speech data and each text in the training label text of any one of the training speech data, respectively determine the training streaming audio data corresponding to each text segmentation result in the multiple text segmentation results, so as to achieve the division of the any one of the training speech data into multiple training streaming audio data, wherein the multiple training streaming audio data include the training streaming audio data corresponding to each text segmentation result;

[0104] The training streaming audio data corresponding to a text segmentation result includes all speech frames corresponding to each text in the corresponding text segmentation result in any of the training speech data.

[0105] In another implementation manner, when the processing unit 402 determines the audio data representation features of each of the plurality of training streaming audio data, the processing unit 402 may be specifically configured to:

[0106] For any training streaming audio data among the multiple training streaming audio data, clustering processing is performed on the any training streaming audio data based on the speech feature of each speech frame in the any training streaming audio data to obtain multiple training clustered speech features of the any training streaming audio data;

[0107] Generate a speech embedding feature for each training clustered speech feature in the plurality of training clustered speech features, and generate a text embedding feature for each text in a text segmentation result corresponding to any training streaming audio data;

[0108] The speech embedding features of each training clustered speech feature and the text embedding features of each text in the text segmentation result corresponding to any training streaming audio data are used to construct the audio data representation features under any training streaming audio data.

[0109] In another embodiment, each text corresponding to a training streaming audio data includes each text in the text segmentation result corresponding to the corresponding training streaming audio data; when the processing unit 402 calculates the model loss value under any training speech data based on the decoding features of each text corresponding to each training streaming audio data, it can be specifically used to:

[0110] For any training streaming audio data among the multiple training streaming audio data, based on the decoding features of each text corresponding to the any training streaming audio data and each text corresponding to the any training streaming audio data, calculate the model loss value under the any training streaming audio data;

[0111] After obtaining the model loss value under each training streaming audio data, a weighted sum is performed on the model loss value under each training streaming audio data to obtain the model loss value under any training speech data.

[0112] In another implementation, the processing unit 402 may also be used to:

[0113] Calling the initial pre-training model, respectively performing feature extraction on each of the training speech data, and obtaining a training feature extraction result for each of the training speech data, wherein the training feature extraction result of one training speech data includes speech features of each speech frame in the corresponding training speech data;

[0114] Using the training feature extraction results of each training speech data, calculate the pre-training model loss value;

[0115] The initial pre-trained model is optimized in the direction of reducing the loss value of the pre-trained model to obtain a second pre-trained model; and based on the second pre-trained model, the first pre-trained model is determined; wherein the target pre-trained model is determined based on the first pre-trained model.

[0116] According to one embodiment of the present invention, Figure 1 or Figure 2 Each step involved in the method shown can be Figure 4 The speech recognition device shown in FIG. 1 is executed by each unit in the speech recognition device shown in FIG. Figure 1 The step S101 shown in FIG. 1 can be performed by Figure 4 The acquisition unit 401 shown in FIG. 1 is executed, and steps S102-S105 can be performed by Figure 4 The processing unit 402 shown in is executed. For example, Figure 2 The steps S201 and S206 shown in FIG. 1 can be performed by Figure 4 The acquisition unit 401 shown in FIG. 1 is executed, and steps S202-S205 and S207-S210 can be performed by Figure 4 The processing unit 402 shown in executes, and so on.

[0117] According to another embodiment of the present invention, Figure 4 Each unit in the speech recognition device shown can be separately or completely combined into one or several other units to constitute, or one (some) of the units can also be split into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present invention. The above-mentioned units are divided based on logical functions. In practical applications, the function of one unit can also be realized by multiple units, or the functions of multiple units can be realized by one unit. In other embodiments of the present invention, any speech recognition device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0118] According to another embodiment of the present invention, the program can be executed by running a program on a general electronic device such as a computer including a central processing unit (CPU), a random access memory medium (RAM), a read-only memory medium (ROM), and other processing elements and storage elements. Figure 1 or Figure 2 A computer program (including program code) for each step involved in the corresponding method shown in Figure 4 The speech recognition device shown in the embodiment of the present invention is used to implement the speech recognition method of the embodiment of the present invention. The computer program can be recorded on a computer storage medium, for example, and loaded into the above-mentioned electronic device through the computer storage medium and run therein.

[0119] After acquiring the target voice data, the embodiment of the present invention can call the target pre-training model to extract the features of the target voice data to obtain the voice features of each target voice frame in the target voice data; then, based on the voice features of each target voice frame, the target voice data can be clustered to obtain multiple clustered voice features; and based on the multiple clustered voice features, at least one text recognition result can be determined; and based on the multiple clustered voice features, at least one text recognition result can be determined. Correspondingly, when the target voice data meets the streaming audio data recognition condition, the audio data representation feature under the target voice data is determined based on the multiple clustered voice features and at least one text recognition result; the target attention model is called to determine the decoding feature of each text recognition result in at least one text recognition result based on the audio data representation feature under the target voice data; and based on the decoding feature of each text recognition result, the target voice recognition result of the target voice data is determined. It can be seen that the embodiment of the present invention can quickly perform voice recognition on the target voice data when it is detected that the target voice data meets the streaming audio data recognition condition, without waiting for the complete voice data to be acquired before starting voice recognition, so as to effectively improve the voice recognition efficiency, and effectively reduce the time delay of voice recognition, thereby improving the user experience.

[0120] Based on the description of the above method embodiment and device embodiment, the exemplary embodiment of the present invention further provides an electronic device, including: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform the method according to the embodiment of the present invention when executed by the at least one processor.

[0121] Exemplary embodiments of the present invention also provide a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present invention.

[0122] An exemplary embodiment of the present invention further provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor of a computer, the computer is used to enable the computer to perform a method according to an embodiment of the present invention.

[0123] refer to Figure 5, a block diagram of an electronic device 500 that can be used as a server or client of the present invention will now be described, which is an example of a hardware device that can be applied to various aspects of the present invention. The electronic device is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0124] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0125] A plurality of components in the electronic device 500 are connected to the I / O interface 505, including: an input unit 506, an output unit 507, a storage unit 508, and a communication unit 509. The input unit 506 may be any type of device capable of inputting information to the electronic device 500, and the input unit 506 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 507 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 508 may include, but is not limited to, a disk, an optical disk. The communication unit 509 allows the electronic device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include, but is not limited to, a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0126] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 501 performs the various methods and processes described above. For example, in some embodiments, the speech recognition method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 500 via ROM 502 and / or a communication unit 509. In some embodiments, the computing unit 501 may be configured to perform the speech recognition method in any other appropriate manner (e.g., by means of firmware).

[0127] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable speech recognition device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0128] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0129] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0130] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0131] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0132] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0133] Furthermore, it should be understood that what is disclosed above is only a preferred embodiment of the present invention, and certainly cannot be used to limit the scope of rights of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A speech recognition method, characterized in that: include: Acquire target voice data, wherein the target voice data includes voice data between a target start time point and a current time point; Calling a target pre-trained model to perform feature extraction on the target speech data to obtain speech features of each target speech frame in the target speech data; Based on the speech features of each target speech frame, clustering the target speech data to obtain a plurality of clustered speech features; and determining at least one text recognition result based on the plurality of clustered speech features; When the target voice data satisfies the streaming audio data recognition condition, determining the audio data representation feature under the target voice data based on the multiple clustered voice features and the at least one text recognition result; Invoking a target attention model to determine a decoding feature of each text recognition result in the at least one text recognition result based on a representation feature of the audio data under the target speech data; And based on the decoding features of each text recognition result, a target speech recognition result of the target speech data is determined.

2. The method according to claim 1, characterized in that: When the target voice data meets the streaming audio data recognition condition, the target voice data is regarded as a streaming audio data, and the method further includes: When an audio input operation is detected, triggering the acquisition of target voice data every preset waiting time; After obtaining the speech recognition result of each stream audio data in the audio data indicated by the audio input operation, completing the speech recognition of the audio data indicated by the audio input operation; Among them, the target starting time point is any one of the following: the audio start time of the audio data indicated by the audio input operation, the end time of the previous streaming audio data, and the next time point after the end time of the previous streaming audio data; when the target voice data meets the streaming audio data recognition conditions, the target starting time point is updated to obtain the starting time point of the next target voice data.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: Acquire a training speech data set, and acquire a training label text of each training speech data in the training speech data set, wherein a training label text includes one or more target boundary markers; For any training speech data in the training speech data set, calling the first pre-training model, performing feature extraction on the any training speech data, and obtaining speech features of each speech frame in the any training speech data; and based on the speech features of each speech frame in the any training speech data, determining the correspondence between each speech frame in the any training speech data and each text in the training label text of the any training speech data; Based on the correspondence between each speech frame in the any training speech data and each text in the training label text of the any training speech data, dividing the any training speech data into a plurality of training streaming audio data, one training streaming audio data corresponds to at least one text in the training label text of the any training speech data; Determine the audio data representation features of each training streaming audio data in the multiple training streaming audio data respectively, and call the initial attention model to generate the decoding features of each text corresponding to each training streaming audio data based on the audio data representation features of each training streaming audio data respectively, so as to calculate the model loss value under any training speech data based on the decoding features of each text corresponding to each training streaming audio data; After obtaining the model loss value under each training speech data, the target model loss value is determined, and the model parameters in the initial attention model are optimized in the direction of reducing the target model loss value to obtain the initial attention model after model optimization, so as to determine the target attention model based on the initial attention model after model optimization.

4. The method according to claim 3, characterized in that The step of dividing any one of the training speech data into a plurality of training streaming audio data based on the correspondence between each speech frame in the any one of the training speech data and each text in the training label text of the any one of the training speech data comprises: According to the target boundary marker, the training label text of any training speech data is divided to obtain multiple text division results; Based on the correspondence between each speech frame in any one of the training speech data and each text in the training label text of any one of the training speech data, respectively determine the training streaming audio data corresponding to each text segmentation result in the multiple text segmentation results, so as to achieve the division of any one of the training speech data into multiple training streaming audio data, wherein the multiple training streaming audio data include the training streaming audio data corresponding to each text segmentation result; The training streaming audio data corresponding to a text segmentation result includes all speech frames corresponding to each text in the corresponding text segmentation result in any of the training speech data.

5. The method according to claim 4, characterized in that The step of respectively determining the audio data representation features of each of the plurality of training streaming audio data comprises: For any training streaming audio data among the multiple training streaming audio data, clustering processing is performed on the any training streaming audio data based on the speech feature of each speech frame in the any training streaming audio data to obtain multiple training clustered speech features of the any training streaming audio data; Generate a speech embedding feature for each training clustered speech feature in the plurality of training clustered speech features, and generate a text embedding feature for each text in a text segmentation result corresponding to any training streaming audio data; The speech embedding features of each training clustered speech feature and the text embedding features of each text in the text segmentation result corresponding to any training streaming audio data are used to construct the audio data representation features under any training streaming audio data.

6. The method according to claim 3, characterized in that Each text corresponding to a training streaming audio data includes each text in the text segmentation result corresponding to the corresponding training streaming audio data; and the model loss value under any training speech data is calculated based on the decoding features of each text corresponding to each training streaming audio data, including: For any training streaming audio data among the multiple training streaming audio data, based on the decoding features of each text corresponding to the any training streaming audio data and each text corresponding to the any training streaming audio data, calculate the model loss value under the any training streaming audio data; After obtaining the model loss value under each training streaming audio data, a weighted sum is performed on the model loss value under each training streaming audio data to obtain the model loss value under any training speech data.

7. The method according to claim 3, characterized in that The method further comprises: Calling the initial pre-training model, respectively performing feature extraction on each of the training speech data, and obtaining a training feature extraction result for each of the training speech data, wherein the training feature extraction result of one training speech data includes speech features of each speech frame in the corresponding training speech data; Using the training feature extraction results of each training speech data, calculate the pre-training model loss value; The initial pre-trained model is optimized in the direction of reducing the loss value of the pre-trained model to obtain a second pre-trained model; and based on the second pre-trained model, the first pre-trained model is determined; wherein the target pre-trained model is determined based on the first pre-trained model.

8. A speech recognition device, characterized in that: The device comprises: An acquisition unit, configured to acquire target voice data, wherein the target voice data includes voice data between a target start time point and a current time point; A processing unit, used to call a target pre-training model, perform feature extraction on the target speech data, and obtain speech features of each target speech frame in the target speech data; The processing unit is further configured to perform clustering processing on the target speech data based on the speech features of each target speech frame to obtain a plurality of clustered speech features; and determine at least one text recognition result based on the plurality of clustered speech features; The processing unit is further configured to determine, when the target voice data satisfies a streaming audio data recognition condition, an audio data representation feature under the target voice data based on the multiple clustered voice features and the at least one text recognition result; The processing unit is also used to call the target attention model to determine the decoding features of each text recognition result in the at least one text recognition result based on the audio data representation features under the target speech data; and determine the target speech recognition result of the target speech data based on the decoding features of each text recognition result.

9. An electronic device, characterized in that: include: processor; as well as Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to make a computer execute the method according to any one of claims 1-7.

Images