Speech processing method and apparatus, speech recognition method and apparatus, and electronic device and medium

Abstract

The present application belongs to the technical field of speech recognition. Provided are a speech processing method and apparatus, a speech recognition method and apparatus, and an electronic device and a medium. The speech processing method comprises: processing speech data, which is input in real time, so as to obtain first speech data of a first frame number; using a first speech recognition model to recognize first text corresponding to the first speech data; outputting the first text; accumulating a plurality of pieces of first speech data, so as to obtain second speech data of a second frame number; using a second speech recognition model to recognize second text corresponding to the second speech data; and using the second text to update a plurality of pieces of output first text corresponding to the plurality of pieces of first speech data.

Description

Speech processing method, speech recognition method, device, electronic device, and medium CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present disclosure incorporates by reference in its entirety the Chinese Patent Application No. 2024106417962 entitled "Speech processing method, speech recognition method, device, electronic device, and medium" filed on May 22, 2024, which is incorporated by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of speech recognition, and in particular to a speech processing method, a speech recognition method, a device, an electronic device, and a medium. BACKGROUND

[0003] With the continuous development of human-computer interaction, the communication between people and machines is becoming more and more like the communication between people. Automatic speech recognition technology (ASR) is the most natural way of interaction, and ASR can convert input speech data recognition into text data. Current ASR cannot display intermediate recognition results in real time.

[0004] Therefore, how to display intermediate recognition results in real time in the process of speech recognition is one of the current problems to be solved in speech recognition. SUMMARY

[0005] The present application aims to at least solve one of the technical problems existing in the background art. To this end, one object of the present application is to provide a speech processing method, a speech recognition method, a device, an electronic device, and a medium to effectively improve user experience.

[0006] Embodiments of the first aspect of the present application provide a speech processing method. The speech processing method comprises: processing real-time input speech data to obtain first speech data of a first frame number; recognizing first text corresponding to the first speech data using a first speech recognition model; outputting the first text; accumulating a plurality of first speech data to obtain second speech data of a second frame number; recognizing second text corresponding to the second speech data using a second speech recognition model; and updating a plurality of first texts corresponding to the plurality of first speech data that have been output using the second text.

[0007] In the technical solution of the embodiment of the application, the first speech recognition model is used to identify and output the first text corresponding to the first speech data, then the second speech recognition model is used to identify the second text corresponding to the second speech data, and the second text is used to update the plurality of first texts corresponding to the plurality of first speech data that have been output. Since the number of frames of the first speech data is small, the first speech recognition model can be used to output the recognition result in real time. The second speech data contains more complete information than the first data, and the second speech recognition model can be used to improve the accuracy of the output text, thereby improving the recognition accuracy of the speech recognition result. Thus, the speech processing method of the embodiment of the application not only realizes the output of the intermediate streaming result of speech recognition to text, but also realizes the output of the final high-accuracy recognition result, and the use of the first speech data model makes up for the deficiency of the existing speech data model without streaming result, thereby effectively improving the user experience.

[0008] In some embodiments, the size of the first speech recognition model is smaller than the size of the second speech recognition model. The size of the first speech recognition model is smaller than the size of the second speech recognition model, wherein the small model has a fast inference speed, realizes the fast recognition and output of the intermediate result, generates the streaming effect, and finally corrects the overall result with the large model, so that the speech processing method realizes the output of the intermediate streaming result of speech recognition to text, and also realizes the output of the final high-accuracy recognition result.

[0009] In some embodiments, identifying the second text corresponding to the second speech data using the second speech recognition model includes: identifying the second text corresponding to the second speech data using the second speech recognition model based on the second speech data and the first text. Identifying the second text corresponding to the second speech data using the second speech recognition model based on the second speech data and the first text can improve the recognition efficiency of the second speech model in identifying the second speech data, thereby improving the speech recognition rate of the second speech model.

[0010] In some embodiments, processing the real-time input speech data to obtain the first speech data of the first number of frames includes: performing segmentation processing on the real-time input speech data in chronological order to obtain third speech data of a third number of frames; obtaining fourth speech data of a first predetermined number of frames after the third speech data, and fifth speech data of a second predetermined number of frames before the third speech data; wherein the first speech data includes the third speech data, the fourth speech data, and the fifth speech data. The first speech data can include historical data and future data, which can further improve the recognition accuracy of the speech processing method.

[0011] In some embodiments, the first speech recognition model includes a Distill-Whisper model, and the second speech recognition model includes a Whisper model.

[0012] The embodiment of the second aspect of the application provides a speech recognition method, comprising: collecting user speech data in real time; processing the speech data to obtain first speech data of a first frame number; recognizing first text corresponding to the first speech data by using a first speech recognition model; outputting the first text; accumulating a plurality of first speech data to obtain second speech data of a second frame number; recognizing second text corresponding to the second speech data by using a second speech recognition model; and updating a plurality of first texts corresponding to the plurality of first speech data that have been output by using the second text.

[0013] In the technical solution of the embodiment of the application, the first text corresponding to the first speech data is recognized by using the first speech recognition model and is output, then the second text corresponding to the second speech data is recognized by using the second speech recognition model, and the plurality of first texts corresponding to the plurality of first speech data that have been output are updated by using the second text. Since the first speech data has a small frame number, the first speech recognition model can be used to output the recognition result in real time. The second speech data contains more complete information than the first data, and the second speech recognition model can improve the accuracy of the output text to improve the recognition accuracy of the speech recognition result. Thus, the speech processing method of the embodiment of the application not only realizes the output of the intermediate streaming result of speech recognition to text, but also realizes the output of the final high-accuracy recognition result, and the first speech data model is used to compensate for the deficiency of the existing speech data model that has no streaming result, thereby effectively improving the user experience.

[0014] The embodiment of the third aspect of the application provides a speech processing device, comprising: a first acquisition module configured to process real-time input speech data to obtain first speech data of a first frame number; a first recognition module configured to recognize first text corresponding to the first speech data by using a first speech recognition model; an output module configured to output the first text; a second acquisition module configured to accumulate a plurality of first speech data to obtain second speech data of a second frame number; a second recognition module configured to recognize second text corresponding to the second speech data by using a second speech recognition model; and an update module configured to update a plurality of first texts corresponding to the plurality of first speech data that have been output by using the second text.

[0015] The embodiment of the fourth aspect of the present application provides a speech recognition device. The speech recognition device comprises: a collection module 510, configured to collect user speech data in real time; a first processing module, configured to process the speech data to obtain first speech data of a first frame number; a first speech recognition module, configured to recognize first text corresponding to the first speech data by using a first speech recognition model; a text output module, configured to output the first text; a second processing module, configured to accumulate a plurality of first speech data to obtain second speech data of a second frame number; a second speech recognition module, configured to recognize second text corresponding to the second speech data by using a second speech recognition model; and a text updating module, configured to update a plurality of first texts corresponding to the plurality of first speech data that have been output by using the second text.

[0016] The embodiment of the fifth aspect of the present application provides a computing device, comprising at least one processor; and at least one memory connected with the at least one processor in communication, the at least one memory storing instructions, which, when executed by the at least one processor alone or jointly, cause the computing device to perform the speech processing method or the speech recognition method of any one of the above.

[0017] The embodiment of the sixth aspect of the present application provides a computer readable storage medium, storing instructions, which, when executed by one or more processors of a computing device alone or jointly, cause the computing device to perform the speech processing method or the speech recognition method of any one of the above.

[0018] The embodiment of the seventh aspect of the present application provides a computer program product, comprising instructions, which, when executed by one or more processors of a computing device alone or jointly, cause the computing device to perform the speech processing method or the speech recognition method of any one of the above.

[0019] The above description is only a summary of the technical solutions of the present application. In order to enable the technical means of the present application to be more clearly understood, and to be implemented according to the content of the description, and in order to enable the above and other purposes, characteristics and advantages of the present application to be more apparent and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0020] In the drawings, identical reference numerals indicate the same elements or elements having the same function throughout the several drawings. The drawings are not necessarily to scale. It should be understood that the drawings only depict some embodiments in accordance with the present disclosure and should not be considered as limiting the scope of the present application.

[0021] FIG. 1 shows a speech processing method flowchart according to an embodiment of the present application;

[0022] FIG. 2 shows a speech recognition method flowchart one according to an embodiment of the present application;

[0023] FIG. 3 shows a flow chart of a voice recognition method according to an embodiment of the present application;

[0024] FIG. 4 shows a structural block diagram of a voice processing apparatus according to an embodiment of the present application;

[0025] FIG. 5 shows a structural block diagram of a voice recognition apparatus according to an embodiment of the present application;

[0026] FIG. 6 shows a block diagram of a computing device according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] The embodiments of the present application will be described in detail with reference to the drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present application, and therefore should not be construed as limiting the scope of the present application.

[0028] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application; the present specification and claims and the aforementioned description of the drawings, the terms "comprising" and "having" and any variations thereof are intended to cover not exclusively inclusive.

[0029] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "a plurality of" is two or more (including two), unless otherwise explicitly and specifically limited.

[0030] Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily independent or alternative embodiments to each other. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0031] In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are a "or" relationship.

[0032] With the continuous development of human-computer interaction, from the initial keyboard operation, mouse click, to the later touch screen, and now the multi-modal interaction of voice and vision, the communication between people and machines is more and more like the communication between people. Automatic speech recognition (ASR) is the most natural interaction way, which can convert the input speech data recognition into text data. ASR has been widely used in industry, household appliances, communication, medical services, and electronic products in various fields. For speech recognition, people have been committed to making it have more accurate recognition accuracy, better environmental adaptability, faster interaction speed, and lower cost.

[0033] In the related art, with the emergence of a new speech recognition algorithm based on Transformer, the recognition rate in the mixed speech scene of multiple languages such as Chinese and English has been greatly improved. However, this algorithm has a fatal problem for user experience, that is, it does not support real-time streaming output of the recognition result, that is, the user can only show the result after the whole sentence recognition is completed, and cannot show the intermediate recognition result in real time.

[0034] Based on this, the embodiment of the present application provides a speech processing method, which uses a first speech recognition model to recognize and output a first text corresponding to first speech data, and then uses a second speech recognition model to recognize a second text corresponding to second speech data, and updates the multiple first texts corresponding to the multiple first speech data that have been output using the second text. Since the number of frames of the first speech data is small, the first speech recognition model can output the recognition result in real time. The second speech data contains more complete information than the first data, and the use of the second speech recognition model can improve the accuracy of the output text to improve the recognition accuracy of the speech recognition result. Thus, the speech processing method of the embodiment of the present application not only realizes the output of the intermediate streaming result of speech recognition to text, but also realizes the output of the final high-accuracy recognition result, and uses the first speech data model to make up for the lack of streaming result of the existing speech data model, effectively improving the user experience.

[0035] The speech processing method disclosed in the embodiment of the present application can be executed in application scenarios such as data processing and human-computer interaction. For example, in the application scenario of data processing, there is a demand for real-time translation and recording of speech data. In this scenario, the speech processing method provided in the embodiment of the present application can be used to recognize and process speech data, thereby converting the speech data into corresponding text data. In the application scenario of human-computer interaction, such as in the intelligent transportation system and the intelligent vehicle-road cooperative system, by recognizing and analyzing the voice instructions of the driver, more accurate reply information can be generated, thereby facilitating navigation guidance.

[0036] According to an aspect of the present application, a speech processing method is provided. FIG. 1 shows a flowchart of a speech processing method according to an embodiment of the present application. As shown in FIG. 1, the speech processing method comprises: step S110, processing real-time input speech data to obtain first speech data of a first frame number; step S120, identifying first text corresponding to the first speech data using a first speech recognition model; step S130, outputting the first text; step S140, accumulating a plurality of first speech data to obtain second speech data of a second frame number; step S150, identifying second text corresponding to the second speech data using a second speech recognition model; and step S160, updating a plurality of first texts corresponding to the plurality of first speech data that have been output using the second text.

[0037] Step S110, processing real-time input speech data to obtain first speech data of a first frame number.

[0038] In an example, the real-time input speech data can be a speech segment obtained by a speech collection device, or a speech segment obtained by real-time acquisition from a speech collection device through a data interface or remote communication transmission. The real-time input speech data can be divided and processed based on the first frame number in chronological order, and the speech data of the first frame number can be obtained as the first speech data for subsequent operations.

[0039] In an example, to further improve recognition accuracy, the first speech data can include historical data and future data.

[0040] Step S120, identifying first text corresponding to the first speech data using a first speech recognition model.

[0041] In an example, after obtaining the first speech data of the first frame number, the first speech data can be input into a speech recognition model for speech recognition processing. The speech recognition model is referred to as the first speech recognition model, and the structure of the first speech recognition model is not limited in the embodiments of the present application. It can be built using any common algorithm in the field of machine learning. Since the first speech recognition model only needs to recognize the first speech data, and the first frame number is usually one or two or a smaller number, the first speech recognition model can be a small-scale speech recognition model. For example, the first speech recognition model can be a Distill-Whisper model.

[0042] Step S130, outputting the first text.

[0043] In an example, the first speech data can be identified using the first speech recognition model to obtain the first text corresponding to the first speech data, i.e., the first text can be output in time.

[0044] In the example, since the first voice data is continuously obtained, the first text is continuously output, and the plurality of first texts are arranged in chronological order. The first text is the initial text in the process of the voice device. The first text can be composed of a single word, of course, it can also be composed of multiple words, and when the first text includes multiple words, the output of the first text can be word by word, or simultaneously output, etc. The first text output by the first voice recognition model can also be inaccurate. Therefore, in the embodiments of the present application, subsequent steps are also used to correct it, so as to improve the recognition accuracy of the obtained text.

[0045] In step S140, a plurality of first voice data are accumulated to obtain second voice data of a second frame number.

[0046] In the example, since the first voice data is continuously obtained, the first voice data can be accumulated, and the plurality of first voice data can be arranged in chronological order to obtain the second voice data. The second frame number is greater than the first frame number, wherein the first frame number and the second frame number can be set according to actual application conditions.

[0047] In step S150, a second voice recognition model is used to recognize a second text corresponding to the second voice data.

[0048] In the example, after obtaining the second voice data of the second frame number, the second voice data can be input into a voice recognition model for voice recognition processing. The voice recognition model is referred to as a second voice recognition model. In the embodiments of the present application, the structure of the second voice recognition model is not specifically limited, and it can be built using any common algorithm in the field of machine learning. Since the second voice recognition model needs to recognize the second voice data, and the second voice data is formed by the plurality of first voice data, the size of the second voice recognition model is greater than that of the first voice recognition model. In order to make the obtained second text more accurate, the second voice recognition model can be a large-scale voice recognition model. For example, the second voice recognition model can be a Whisper model.

[0049] In step S160, the second text is used to update the plurality of first texts corresponding to the plurality of first voice data that have been output.

[0050] In the example, the second text can replace the plurality of first texts that have been output. That is, the second text is the final voice recognition result. The second text is the target text.

[0051] According to the speech processing method provided in the embodiments of the present application, the first speech recognition model is used to recognize the first text corresponding to the first speech data and output the first text, and then the second speech recognition model is used to recognize the second text corresponding to the second speech data, and the second text is used to update the plurality of first texts corresponding to the plurality of first speech data that have been output. Since the number of frames of the first speech data is small, the first speech recognition model can be used to output the recognition result in real time. The second speech data contains more complete information than the first data, and the second speech recognition model can be used to improve the accuracy of the output text, so as to improve the recognition accuracy of the speech recognition result. Thus, the speech processing method provided in the embodiments of the present application not only realizes the output of the intermediate streaming result of speech recognition to text, but also realizes the output of the final high-accuracy recognition result, and the first speech data model is used to make up for the deficiency of the existing speech data model that does not have a streaming result, thereby effectively improving the user experience.

[0052] According to some embodiments of the present application, the size of the first speech recognition model is smaller than the size of the second speech recognition model.

[0053] In examples, the size of a model generally refers to the number of parameters or the number of layers of the model. The number of parameters refers to the total number of connection weights and bias terms in the model, and the more the number of parameters, the larger the capacity of the model and the more complex the representation that can be learned. The number of layers refers to the number of hidden layers in the model, including the input layer and the output layer. The more the number of layers of the model, the richer the feature hierarchy that the model can capture. These two indicators are generally used to measure the complexity and capacity of the model. A larger size model generally requires more computing resources and storage space, so a trade-off needs to be made between hardware resources and efficiency. A larger size model may require more training data for effective training to prevent overfitting and poor generalization performance. Therefore, when selecting a model, the size of the model needs to be balanced according to the specific task and available resources to achieve the best performance and effect.

[0054] In examples, the first speech recognition model is used to recognize the first text corresponding to the first speech data. Since the first speech data is small, the first speech recognition model can be a small-scale speech model that can recognize the first text corresponding to the first speech data, that is, the intermediate text result. At the same time, since the first speech recognition model is small, it can quickly generate the first text, so the generation rate of the first text is fast and can be generated in real time, but the result of the first text is not very accurate.

[0055] In an example, the second speech data corresponding to the second frame number obtained by accumulating the plurality of first speech data is identified by using a second speech recognition model. Since the second speech data is large, the second speech recognition model can select a large-scale speech model, and can identify the second text corresponding to the second speech data, that is, the final text result. Since the second speech recognition model is large, the rate of generating the second text is slow, but the result of the second text is more accurate.

[0056] In the embodiments of the present application, the size of the first speech recognition model is smaller than the size of the second speech recognition model. The small model has a fast inference speed, realizes fast identification and output of the intermediate result, generates a streaming effect, and finally corrects the overall result by using the large model. The speech processing method realizes the output of the intermediate streaming result of speech recognition to text and the output of the final high-accuracy recognition result.

[0057] According to some embodiments of the present application, identifying the second text corresponding to the second speech data by using the second speech recognition model comprises: identifying the second text corresponding to the second speech data by using the second speech recognition model based on the second speech data and the first text.

[0058] In an example, when using the second speech recognition model for speech recognition, the first text (the recognition result of the first speech model) can be used as a prompt word or a hot word for the second speech recognition model, so as to speed up the inference of the second speech recognition model and improve the overall recognition speed.

[0059] In the embodiments of the present application, the second text corresponding to the second speech data is identified by using the second speech recognition model based on the second speech data and the first text. This can improve the recognition efficiency of the second speech model in identifying the second speech data, thereby improving the speech recognition rate of the second speech model.

[0060] According to some embodiments of the present application, the real-time input speech data is processed to obtain first speech data of a first frame number, which comprises: the real-time input speech data is processed by cutting in time sequence to obtain third speech data of a third frame number; fourth speech data of a first predetermined frame number after the third speech data is obtained, and fifth speech data of a second predetermined frame number before the third speech data is obtained; wherein the first speech data comprises the third speech data, the fourth speech data and the fifth speech data.

[0061] For example, the first frame number is 30 frames, wherein the third frame number can be 10 frames, the first predetermined frame number can be 10 frames, and the second predetermined frame number can be 10 frames. After 10 frames of speech data are obtained in real time, the most recent 10 frames of speech data can be obtained from the most recently obtained speech data, and at the same time, 10 frames of speech data are accumulated to form 30 frames of speech data for processing.

[0062] In the embodiments of the present application, the historical data and the future data can be included in the first voice data, which can further improve the recognition accuracy of the voice processing method.

[0063] According to some embodiments of the present application, the first voice recognition model comprises a Distill-Whisper model, and the second voice recognition model comprises a Whisper model.

[0064] In an example, an optional model structure of the first voice recognition model is proposed, which can be a Distill-Whisper model. The Distill-Whisper model is a model optimized and simplified on the core function of the Whisper model, which greatly reduces the scale and greatly improves the speed compared with the Whisper model, and is suitable for application scenarios requiring fast response, such as real-time translation, real-time conference recording, etc. The Distill-Whisper model can be used as an assistant model of the Whisper model for speculative decoding to improve the speed of voice recognition.

[0065] In an example, an optional model structure of the second voice recognition model is proposed, which can be a Whisper model. The Whisper model adopts a simple end-to-end method, i.e. using the encoder-decoder structure of the Transformer model. The input audio is divided into 30-second modules, then converted into log-Mel spectrograms, and then passed to the encoder for attention calculation, and finally the corresponding text is predicted by the decoder. Compared with other voice recognition models, it has higher accuracy.

[0066] According to another aspect of the present application, a voice recognition method is provided. FIG. 2 shows a flowchart of a voice recognition method according to an embodiment of the present application. As shown in FIG. 2, the voice recognition method comprises: step S210, collecting user voice data in real time; step S220, processing the voice data to obtain first voice data of a first frame number; step S230, identifying the first text corresponding to the first voice data using the first voice recognition model; step S240, outputting the first text; step S250, accumulating a plurality of first voice data to obtain second voice data of a second frame number; step S260, identifying the second text corresponding to the second voice data using the second voice recognition model; and step S270, updating the plurality of first texts corresponding to the plurality of first voice data that have been outputted using the second text.

[0067] In practical applications, the speech recognition method of the embodiments of the present application can be applied to various scenarios, such as human-computer interaction, intelligent question answering, voice assistants, conferences, live broadcasts, and other scenarios that require caption output. The technical solutions of the embodiments of the present application can be executed by a server, and the output of the first text and the updating of the output result using the second text can be that the server sends the first text to the corresponding client, the client displays the first text, and the server sends the second text to the client to instruct the client to update its display result. Of course, the client can also execute the technical solutions.

[0068] In addition, in a local integrated scenario that requires speech recognition, the technical solutions of the embodiments of the present application can also be executed by a local integrated machine. The local integrated machine can realize recognition and output processing, and can realize speech collection, recognition, and real-time output of recognized text without networking. Local integrated machines are widely used in banks, courts, and other scenarios to provide self-service for users.

[0069] For example, FIG. 3 shows a flowchart of a speech recognition method according to an embodiment of the present application. As shown in FIG. 3, step S310, real-time collection of user speech data input by a user through a microphone, the user speech data including "What's the weather like in Beijing today?"; step S320, processing of the speech data to obtain first speech data of a first frame number, wherein the first frame number is 1; step S330, identification of first text corresponding to the first speech data using a first speech recognition model, wherein the size of the first speech recognition model is smaller than the size of a second speech recognition model, and the first speech recognition model can be a Distill-Whisper model; step S340, output of the first text, wherein since the first speech data is continuously obtained, the first text is also continuously output, and multiple first texts are arranged in chronological order. The first text can be composed of single characters, and the output of the first text can be character-by-character output; step S350, accumulation of multiple first speech data to obtain second speech data of a second frame number, the second speech data being "What's the weather like in Beijing today?"; step S360, identification of second text corresponding to the second speech data using a second speech recognition model, wherein the size of the second speech recognition model is larger than the size of the first speech recognition model, the second speech recognition model can be a Whisper model, and the second text is "What's the weather like in Beijing today?"; and step S370, updating of multiple first texts corresponding to multiple first speech data that have been output using the second text.

[0070] In the embodiments of the present application, the first speech recognition model is used to recognize the first text corresponding to the first speech data and output, then the second speech recognition model is used to recognize the second text corresponding to the second speech data, and the second text is used to update the multiple first texts corresponding to the multiple first speech data which have been output. Since the number of frames of the first speech data is small, the first speech recognition model can be used to output the recognition result in real time. The second speech data contains more complete information than the first data, and the second speech recognition model can improve the accuracy of the output text to improve the recognition accuracy of the speech recognition result. Thus, the speech processing method of the embodiments of the present application not only realizes the output of the intermediate streaming result of speech recognition to text, but also realizes the output of the final high-accuracy recognition result, and uses the first speech data model to make up for the deficiency of the existing speech data model without streaming result, effectively improving the user experience.

[0071] According to another aspect of the present application, a speech processing apparatus is provided. FIG. 4 shows a structural block diagram of the speech processing apparatus according to the embodiments of the present application. As shown in FIG. 4, the speech processing apparatus comprises: a first obtaining module 410 configured to process the real-time input speech data to obtain first speech data with a first number of frames; a first recognition module 420 configured to recognize first text corresponding to the first speech data by using a first speech recognition model; an output module 430 configured to output the first text; a second obtaining module 440 configured to accumulate multiple first speech data to obtain second speech data with a second number of frames; a second recognition module 450 configured to recognize second text corresponding to the second speech data by using a second speech recognition model; and an updating module 460 configured to update the multiple first texts corresponding to the multiple first speech data which have been output by using the second text.

[0072] Since the first obtaining module 410, the first recognition module 420, the output module 430, the second obtaining module 440, the second recognition module 450, and the updating module 460 used in the speech processing apparatus can correspond to steps S110, S120, S130, S140, S150, and S160 of FIG. 1 respectively, details of each aspect will not be repeated here.

[0073] In some embodiments, the second recognition module 450 is further configured to recognize the second text corresponding to the second speech data by using the second speech recognition model based on the second speech data and the first text.

[0074] In some embodiments, the first obtaining module 410 is further configured to split the real-time input voice data in time sequence to obtain third voice data of a third frame number; obtain fourth voice data of a first predetermined frame number after the third voice data, and fifth voice data of a second predetermined frame number before the third voice data; wherein the first voice data comprises the third voice data, the fourth voice data and the fifth voice data.

[0075] According to the embodiments of the present application, the first voice recognition model is used to recognize the first text corresponding to the first voice data and output, and then the second voice recognition model is used to recognize the second text corresponding to the second voice data, and the second text is used to update the multiple first texts corresponding to the multiple first voice data which have been output. Since the frame number of the first voice data is small, the first voice recognition model can be used to output the recognition result in real time. The second voice data contains more complete information than the first data, and the second voice recognition model can be used to improve the accuracy of the output text, so as to improve the recognition accuracy of the voice recognition result. Thus, the voice processing method of the embodiments of the present application not only realizes the output of the intermediate streaming result of voice recognition to text, but also realizes the output of the final high-accuracy recognition result, and the first voice data model makes up for the deficiency of the existing voice data model without streaming result, effectively improving the user experience.

[0076] According to another aspect of the present application, a voice recognition device is provided. FIG. 5 shows a structural block diagram of the voice recognition device according to the embodiments of the present application. As shown in FIG. 5, the voice recognition device comprises: a collection module 510 configured to collect user voice data in real time; a first processing module 520 configured to process the voice data to obtain first voice data of a first frame number; a first voice recognition module 530 configured to use a first voice recognition model to recognize first text corresponding to the first voice data; a text output module 540 configured to output the first text; a second processing module 550 configured to accumulate multiple first voice data to obtain second voice data of a second frame number; a second voice recognition module 560 configured to use a second voice recognition model to recognize second text corresponding to the second voice data; and a text updating module 570 configured to use the second text to update multiple first texts corresponding to multiple first voice data which have been output.

[0077] Since the collection module 510, the first processing module 520, the first voice recognition module 530, the text output module 540, the second processing module 550, the second voice recognition module 560 and the text updating module 570 used in the voice recognition device can correspond to the steps S210, S220, S230, S240, S250, S260 and S270 of FIG. 1 respectively, the details of each aspect will not be repeated here.

[0078] According to the embodiments of the present application, the first speech data is recognized by the first speech recognition model to output the first text corresponding to the first speech data, and then the second speech data is recognized by the second speech recognition model to output the second text corresponding to the second speech data, and the second text is used to update the first texts corresponding to the first speech data. Since the number of frames of the first speech data is small, the first speech recognition model can be used to output the recognition result in real time. The second speech data contains more complete information than the first data, and the second speech recognition model can improve the accuracy of the output text to improve the recognition accuracy of the speech recognition result. Thus, the speech processing method of the embodiments of the present application not only realizes the output of the intermediate streaming result of speech recognition to text, but also realizes the output of the final high-accuracy recognition result, and the first speech data model makes up for the lack of the existing speech data model without streaming result, effectively improving the user experience.

[0079] According to another aspect of the present application, a computing device is also provided, including at least one processor; and at least one memory communicatively connected with the at least one processor, the at least one memory storing instructions which, when executed by the at least one processor alone or jointly, cause the computing device to perform the speech recognition method or the speech processing method of any one of the preceding embodiments.

[0080] FIG. 6 shows a block diagram of a computing device according to an embodiment of the present application.

[0081] For example, the speech processing apparatus and the speech recognition apparatus described above can be implemented in whole or at least in part by the computing device 600 or similar devices or systems.

[0082] The computing device 600 can be various different types of devices. Examples of the computing device 600 include, but are not limited to, an industrial control computer, a desktop computer, a server computer, a notebook or netbook computer, a mobile device (e.g., a tablet computer, a cellular or other wireless telephone (e.g., a smartphone), a notepad computer, a mobile station), and the like.

[0083] The computing device 600 can include at least one processor 602, a memory 604, a communication interface(s) 606, a display device 608, other input / output (I / O) devices 610, and one or more mass storage devices 612, which are capable of communicating with each other, such as over a system bus 614 or other appropriate connection.

[0084] The processor 602 can be a single processing unit or a plurality of processing units, all of which can include single or multiple computing units or multiple cores. The processor 602 can be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and / or any devices that manipulate signals based on operational instructions. Among other capabilities, the processor 602 can be configured to fetch and execute computer-readable instructions stored in the memory 604, the mass storage device 612, or any other computer-readable medium.

[0085] The memory 604 and the mass storage device 612 are examples of computer-readable storage media for storage of instructions which are executed by the processor 602 to implement the various functionalities described above. By way of example, the memory 604 can generally include both volatile memory and nonvolatile memory (e.g., RAM, ROM, etc.). Further, the mass storage device 612 can generally include hard disk drives, solid state drives, removable media, including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CD or DVD), storage arrays, network attached storage, storage area networks, etc. Both the memory 604 and the mass storage device 612 can be collectively referred to herein as the memory or computer-readable storage medium, and can be non-transitory media capable of storing the computer-readable, processor-executable program instructions as computer program code which can be executed by the processor 602 as a particular machine configured to implement the operations and functionalities described in the examples herein.

[0086] A plurality of programs can be stored on the mass storage device 612. These programs include an operating system 616, one or more application programs 618, other programs 620, and program data 622, and they can be loaded into the memory 604 for execution. Examples of such application programs or program modules can include, for example, computer program logic (e.g., computer program code or instructions) for implementing the embodiments described herein.

[0087] While illustrated in FIG. 6 as being stored in the memory 604 of the computing device 600, the operating system 616, the application programs 618, the other programs 620, and the program data 622, or portions thereof, can be implemented in any form of computer-readable media accessible by the computing device 600. As used herein, “computer-readable media” includes both computer-readable storage media and communication media.

[0088] Computer-readable storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer-readable storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD), or other optical storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information for access by a computing device. In contrast, communication media can embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal, such as a carrier wave or other transport mechanism. As defined herein, computer-readable storage media does not include communication media.

[0089] One or more communication interfaces 606 are used to exchange data with other devices, such as over a network, direct connection, or the like. Such communication interfaces can be one or more of: any type of network interface (e.g., network interface card (NIC)), wired or wireless (such as IEEE 802.11 wireless LAN (WLAN)) wireless interface, Worldwide Interoperability for Microwave Access (Wi-MAX) interface, Ethernet interface, Universal Serial Bus (USB) interface, cellular network interface, Bluetooth™ interface, near field communication (NFC) interface, or the like. The communication interfaces 606 can facilitate communications within a variety of networks and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, satellite, etc.), the Internet, and the like. The communication interfaces 606 can also provide communication with external storage devices (not shown), such as storage arrays, network attached storage, storage area networks, and the like.

[0090] In some examples, a display device 608, such as a monitor, can be included for displaying information and images to a user. Other I / O devices 610 can be devices that receive various inputs from a user and provide various outputs to the user, and can include touch input devices, gesture input devices, cameras, keyboards, remote controls, mice, printers, audio input / output devices, and the like.

[0091] The techniques described herein can be supported by these various configurations of the computing device 600 and are not limited to the specific examples carried through in this portion of the specification. For example, the functionality can also be implemented all or in part through the use of distributed systems, such as over the "cloud." Cloud includes and / or comprises a platform of resources. The platform abstracts underlying functionality of hardware (e.g., servers) and software resources of the cloud. Resources can include applications and / or data that is stored, accessed, and / or executed via the cloud over the Internet, for example. The resources can also include services that are offered over the Internet and / or through a subscriber network, such as a cellular or Wi-Fi network. The platform can abstract resources and functionality of the cloud to connect computing device 600 with other computing devices. For example, the functionality can be implemented partially on the computing device 600 and partially by the platform that abstracts the functionality of the cloud.

[0092] According to another aspect of the present application, there is also provided a computer readable storage medium having stored thereon a computer program, which, when executed by a processor, implements the speech recognition method or the speech processing method of any of the preceding embodiments.

[0093] According to another aspect of the present application, there is also provided a computer program product comprising instructions which, when executed by one or more processors of a computing device, singly or jointly cause the computing device to perform the speech recognition method or the speech processing method of any of the preceding embodiments.

[0094] It should be understood that, although the various steps of the flowcharts involved in the above embodiments are shown in sequence according to the direction of the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise explicitly stated herein, the execution of these steps is not strictly limited in sequence, and these steps can be executed in other orders. Moreover, as described above, at least some of the steps of the flowcharts involved in the above embodiments can include multiple steps or stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least some of the other steps or the steps or stages in other steps.

[0095] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them. Although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some or all of the technical features can be replaced equivalently. Such modifications or replacements do not change the essence of the corresponding technical solutions, which should be covered in the scope of the claims and the specification of the present application. In particular, the technical features mentioned in each embodiment can be combined in any manner as long as there is no structural conflict. The present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A voice processing method, characterized by, The method comprises: processing real-time input voice data to obtain first voice data of a first frame number; recognizing first text corresponding to the first voice data by using a first voice recognition model; outputting the first text; accumulating a plurality of the first voice data to obtain second voice data of a second frame number; recognizing second text corresponding to the second voice data by using a second voice recognition model; updating a plurality of first texts corresponding to the plurality of first voice data that have been output by using the second text.

2. The method of claim 1, wherein, The size of the first voice recognition model is smaller than the size of the second voice recognition model.

3. The method according to claim 1 or 2, characterized in that, The recognizing second text corresponding to the second voice data by using a second voice recognition model comprises: recognizing second text corresponding to the second voice data by using a second voice recognition model based on the second voice data and the first text.

4. The method of claim 1, wherein, The processing real-time input voice data to obtain first voice data of a first frame number comprises: segmenting and processing real-time input voice data in chronological order to obtain third voice data of a third frame number; obtaining fourth voice data of a first predetermined frame number after the third voice data and fifth voice data of a second predetermined frame number before the third voice data; wherein the first voice data comprises the third voice data, the fourth voice data and the fifth voice data.

5. The method according to claim 1 or 2, characterized in that, The first voice recognition model comprises a Distill-Whisper model, and the second voice recognition model comprises a Whisper model.

6. A voice recognition method characterized by comprising: The method comprises: real-time collection of user voice data; processing the voice data to obtain first voice data of a first frame number; recognizing first text corresponding to the first voice data by using a first voice recognition model; outputting the first text; accumulating a plurality of the first voice data to obtain second voice data of a second frame number; recognizing second text corresponding to the second voice data by using a second voice recognition model; updating a plurality of first texts corresponding to the plurality of first voice data that have been output by using the second text.

7. A speech processing device, characterized by The method comprises: a first acquisition module configured to process real-time input voice data to obtain first voice data of a first frame number; a first recognition module configured to recognize first text corresponding to the first voice data by using a first voice recognition model; an output module configured to output the first text; a second acquisition module configured to accumulate a plurality of the first voice data to obtain second voice data of a second frame number; a second recognition module configured to recognize second text corresponding to the second voice data by using a second voice recognition model; an updating module configured to update a plurality of first texts corresponding to the plurality of first voice data that have been output by using the second text.

8. A speech recognition apparatus characterized by comprising: The method comprises: a collection module configured to real-time collection of user voice data; a first processing module configured to process the voice data to obtain first voice data of a first frame number; a first voice recognition module configured to recognize first text corresponding to the first voice data by using a first voice recognition model; a text output module configured to output the first text; The second processing module is configured to accumulate the plurality of first voice data to obtain second voice data of a second frame number; The second voice recognition module is configured to recognize the second voice data by using a second voice recognition model to obtain corresponding second text; The text updating module is configured to update the plurality of first texts corresponding to the plurality of first voice data that have been output by using the second text.

9. A computing device, comprising: Comprise: At least one processor; And At least one memory connected with the at least one processor in communication, the at least one memory stores instructions, when the instructions are executed by the at least one processor alone or jointly, make the computing device execute the speech processing method in any one of claims 1 to 5 or the speech recognition method in claim 6.

10. A computer-readable storage medium, characterized in that, Store instructions, when the instructions are executed by one or more processors of a computing device alone or jointly, make the computing device execute the speech processing method in any one of claims 1 to 5 or the speech recognition method in claim 6.

11. A computer program product, characterised in that, Include instructions, when the instructions are executed by one or more processors of a computing device alone or jointly, make the computing device execute the speech processing method in any one of claims 1 to 5 or the speech recognition method in claim 6.

Images