Speech processing method and device, speech recognition method and device, electronic equipment and medium

Abstract

The invention provides a voice processing method and device, a voice recognition method and device, electronic equipment and a medium, and belongs to the technical field of voice recognition. The voice processing method comprises the following steps: processing voice data input in real time to obtain first voice data of a first frame number; recognizing a first text corresponding to the first voice data by using a first voice recognition model; outputting the first text; accumulating the plurality of first voice data to obtain second voice data of a second frame number; recognizing a second text corresponding to the second voice data by using a second voice recognition model; and updating the plurality of output first texts corresponding to the plurality of pieces of first voice data by using the second text.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a speech processing method, a speech recognition method, a device, an electronic device and a medium. Background Art

[0002] With the continuous development of human-computer interaction, the communication between people and machines is becoming more and more like the communication between people. Automatic speech recognition technology (ASR) is the most natural way of interaction. ASR can recognize and convert the input voice data into text data. The current ASR cannot display the intermediate recognition results in real time.

[0003] Therefore, how to display the intermediate recognition results in real time during the speech recognition process is one of the current problems that need to be solved urgently in speech recognition. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems in the background technology. To this end, one object of the present application is to provide a speech processing method, a speech recognition method, a device, an electronic device and a medium to effectively improve the user experience.

[0005] The embodiment of the first aspect of the present application provides a speech processing method. The speech processing method includes: processing the real-time input speech data to obtain the first speech data of the first frame number; using the first speech recognition model to recognize the first text corresponding to the first speech data; outputting the first text; accumulating multiple first speech data to obtain the second speech data of the second frame number; using the second speech recognition model to recognize the second text corresponding to the second speech data; using the second text to update the multiple first texts corresponding to the multiple first speech data that have been output.

[0006] In the technical solution of the embodiment of the present application, a first speech recognition model is used to recognize and output the first text corresponding to the first speech data, and then a second speech recognition model is used to recognize the second text corresponding to the second speech data, and the second text is used to update the output multiple first texts corresponding to the multiple first speech data. Since the number of frames of the first speech data is small, the recognition result can be output in real time using the first speech recognition model. The second speech data contains more complete information than the first data, and the second speech recognition model can be used to improve the accuracy of the output text, so as to improve the recognition accuracy of the speech recognition result. As a result, the speech processing method of the embodiment of the present application not only realizes the output of the intermediate streaming results of speech recognition to text, but also realizes the output of the final high-accuracy recognition results, and the use of the first speech data model makes up for the deficiency of the existing speech data model that has no streaming results, effectively improving the user experience.

[0007] In some embodiments, the scale of the first speech recognition model is smaller than that of the second speech recognition model. The scale of the first speech recognition model is smaller than that of the second speech recognition model, wherein the small model has a fast inference speed, realizes rapid recognition and output of intermediate results, generates a streaming effect, and finally uses the large model to correct the overall result, so that the speech processing method realizes both the output of intermediate streaming results from speech recognition to text and the output of the final recognition result with high accuracy.

[0008] In some embodiments, using the second speech recognition model to recognize the second text corresponding to the second speech data includes: based on the second speech data and the first text, using the second speech recognition model to recognize the second text corresponding to the second speech data. Using the second speech recognition model to recognize the second text corresponding to the second speech data based on the second speech data and the first text can improve the recognition efficiency of the second speech model in recognizing the second speech data, thereby improving the speech recognition rate of the second speech model.

[0009] In some embodiments, processing the real-time input voice data to obtain the first voice data of the first frame number includes: segmenting the real-time input voice data in chronological order to obtain the third voice data of the third frame number; obtaining the fourth voice data of the first predetermined frame number after the third voice data, and the fifth voice data of the second predetermined frame number before the third voice data; wherein the first voice data includes the third voice data, the fourth voice data and the fifth voice data. The first voice data may include historical data and future data, which can further improve the recognition accuracy of the voice processing method.

[0010] In some embodiments, the first speech recognition model includes a Distill-Whisper model and the second speech recognition model includes a Whisper model.

[0011] The embodiment of the second aspect of the present application provides a speech recognition method including: real-time collection of user voice data; processing the voice data to obtain first voice data of a first frame number; using a first speech recognition model to recognize a first text corresponding to the first voice data; outputting the first text; accumulating multiple first voice data to obtain second voice data of a second frame number; using a second speech recognition model to recognize a second text corresponding to the second voice data; and using the second text to update the output multiple first texts corresponding to the multiple first voice data.

[0012] In the technical solution of the embodiment of the present application, a first speech recognition model is used to recognize and output the first text corresponding to the first speech data, and then a second speech recognition model is used to recognize the second text corresponding to the second speech data, and the second text is used to update the output multiple first texts corresponding to the multiple first speech data. Since the number of frames of the first speech data is small, the recognition result can be output in real time using the first speech recognition model. The second speech data contains more complete information than the first data, and the second speech recognition model can be used to improve the accuracy of the output text, so as to improve the recognition accuracy of the speech recognition result. As a result, the speech processing method of the embodiment of the present application not only realizes the output of the intermediate streaming results of speech recognition to text, but also realizes the output of the final high-accuracy recognition results, and the use of the first speech data model makes up for the deficiency of the existing speech data model that has no streaming results, effectively improving the user experience.

[0013] An embodiment of the third aspect of the present application provides a speech processing device, which includes: a first acquisition module, configured to process real-time input speech data to obtain a first frame number of first speech data; a first recognition module, configured to use a first speech recognition model to recognize a first text corresponding to the first speech data; an output module, configured to output the first text; a second acquisition module, configured to accumulate multiple first speech data to obtain a second frame number of second speech data; a second recognition module, configured to use a second speech recognition model to recognize a second text corresponding to the second speech data; and an update module, configured to use the second text to update the output multiple first texts corresponding to the multiple first speech data.

[0014] The embodiment of the fourth aspect of the present application provides a speech recognition device. The speech recognition device includes: a collection module 510, configured to collect user speech data in real time; a first processing module, configured to process the speech data to obtain a first frame number of first speech data; a first speech recognition module, configured to use a first speech recognition model to recognize a first text corresponding to the first speech data; a text output module, configured to output the first text; a second processing module, configured to accumulate multiple first speech data to obtain a second frame number of second speech data; a second speech recognition module, configured to use a second speech recognition model to recognize the second speech data, corresponding to the second text; a text update module, configured to use the second text to update the output multiple first texts corresponding to the multiple first speech data.

[0015] An embodiment of the fifth aspect of the present application provides a computing device, comprising at least one processor; and at least one memory communicatively connected to the at least one processor, wherein the at least one memory stores instructions, which, when executed individually or collectively by the at least one processor, enable the computing device to execute any one of the above-mentioned speech processing methods or speech recognition methods.

[0016] An embodiment of the sixth aspect of the present application provides a computer-readable storage medium storing instructions, which, when executed individually or collectively by one or more processors of a computing device, cause the computing device to execute any of the above-mentioned speech processing methods or speech recognition methods.

[0017] An embodiment of the seventh aspect of the present application provides a computer program product, comprising instructions, which, when executed individually or collectively by one or more processors of a computing device, cause the computing device to execute any of the above-mentioned speech processing methods or speech recognition methods.

[0018] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments disclosed in the present application and should not be regarded as limiting the scope of the present application.

[0020] Figure 1 A flow chart of a speech processing method according to an embodiment of the present application is shown;

[0021] Figure 2 The speech recognition method according to the embodiment of the present application is shown in FIG. Figure 1 ;

[0022] Figure 3 The speech recognition method according to the embodiment of the present application is shown in FIG. Figure 2 ;

[0023] Figure 4 A structural block diagram of a speech processing device according to an embodiment of the present application is shown;

[0024] Figure 5 A structural block diagram of a speech recognition device according to an embodiment of the present application is shown;

[0025] Figure 6 A block diagram of a computing device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] The following embodiments of the technical solution of the present application are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application; the terms "including" and "having" in the specification and claims of this application and the above-mentioned figure descriptions and any variations thereof are intended to cover non-exclusive inclusions.

[0028] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is more than two (including two), unless otherwise clearly and specifically defined.

[0029] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0030] In the description of the embodiments of the present application, the term "and / or" is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0031] With the continuous development of human-computer interaction, from the initial keyboard operation and mouse click, to the later touch screen, and now the multi-modal interaction combining voice and vision, the communication between people and machines is becoming more and more like the communication between people. Automatic speech recognition technology (Automatic Speech Recognition, ASR) is the most natural way of interaction. ASR can recognize the input voice data and convert it into text data. ASR is currently widely used in various fields such as industry, home appliances, communications, medical services, and electronic products. For speech recognition, people have been committed to making it have more accurate recognition accuracy, better environmental adaptability, faster interaction speed, and lower cost.

[0032] In related technologies, with the emergence of new speech recognition algorithms based on Transformer, the recognition rate has been greatly improved in scenarios where multiple languages ​​are mixed, such as Chinese and English. However, this algorithm also has a fatal problem for user experience. It does not support real-time streaming output of recognition results. In other words, users can only display the results after the recognition of a whole sentence is completed, and cannot display the intermediate recognition results in real time.

[0033] Based on this, an embodiment of the present application provides a speech processing method, which uses a first speech recognition model to recognize and output the first text corresponding to the first speech data, and then uses a second speech recognition model to recognize the second text corresponding to the second speech data, and uses the second text to update the output multiple first texts corresponding to the multiple first speech data. Since the number of frames of the first speech data is small, the recognition result can be output in real time using the first speech recognition model. The second speech data contains more complete information than the first data, and the second speech recognition model can be used to improve the accuracy of the output text to improve the recognition accuracy of the speech recognition result. As a result, the speech processing method of the embodiment of the present application not only realizes the output of the intermediate streaming results of speech recognition to text, but also realizes the output of the final high-accuracy recognition results, and uses the first speech data model to make up for the lack of streaming results in the existing speech data model, effectively improving the user experience.

[0034] The speech processing method disclosed in the embodiment of the present application can be executed in application scenarios such as data processing and human-computer interaction. For example, in the application scenario of data processing, there is a need to translate and record voice data in real time. In this scenario, the speech processing method provided in the embodiment of the present application can be used to recognize and process the voice data, thereby converting the voice data into corresponding text data. In the application scenarios of human-computer interaction, such as in intelligent transportation systems and intelligent vehicle-road cooperative systems, by recognizing and analyzing the driver's voice commands, it can help to generate more accurate response information, thereby facilitating the provision of navigation guidance.

[0035] According to one aspect of the present application, a speech processing method is provided. Figure 1 FIG. 1 shows a flow chart of a speech processing method according to an embodiment of the present application. Figure 1As shown, the speech processing method includes: step S110, processing the real-time input speech data to obtain the first speech data of the first frame number; step S120, using the first speech recognition model to recognize the first text corresponding to the first speech data; step S130, outputting the first text; step S140, accumulating multiple first speech data to obtain the second speech data of the second frame number; step S150, using the second speech recognition model to recognize the second text corresponding to the second speech data; step S160, using the second text to update the output multiple first texts corresponding to the multiple first speech data.

[0036] Step S110: Process the voice data input in real time to obtain first voice data of a first number of frames.

[0037] In the example, the real-time input voice data may be a voice segment acquired through a voice acquisition device, or a voice segment acquired from a voice acquisition device in real time through a data interface or remote communication transmission. The real-time input voice data may be segmented and processed based on the first frame number in chronological order, and subsequent operations may be performed as the first voice data for each accumulated first frame number of voice data.

[0038] In the example, in order to further improve the recognition accuracy, the first voice data may include historical data and future data.

[0039] Step S120: using a first speech recognition model to recognize a first text corresponding to the first speech data.

[0040] In the example, after obtaining the first voice data of the first frame number, the first voice data can be input into a voice recognition model for voice recognition processing. The voice recognition model here is recorded as the first voice recognition model. In the embodiment of the present application, there is no specific restriction on the structure of the first voice recognition model, and it can be built using any commonly used algorithm in the field of machine learning. Since the first voice recognition model only needs to recognize the first voice data, and the first number of frames can usually be a smaller number such as 1 or two, the first voice recognition model can be a small-scale voice recognition model. Exemplarily, the first voice recognition model can be a Distill-Whisper model.

[0041] Step S130: output the first text.

[0042] In the example, the first speech data can be recognized using the first speech recognition model to obtain the first text corresponding to the first speech data, that is, the first text can be output in time.

[0043] In the example, since the first voice data will be continuously obtained, the first text will also be continuously output, and multiple first texts will be arranged in chronological order. The first text is the initial text in the voice device process. The first text can be composed of a single word, of course, it can also be composed of multiple words. When the first text includes multiple words, the output of the first text can be output word by word, or it can be output simultaneously, etc. The first text output by the first speech recognition model may also be inaccurate. Therefore, in the embodiment of the present application, it is also corrected through subsequent steps to improve the recognition accuracy of the obtained text.

[0044] Step S140: Accumulate a plurality of first voice data to obtain a second number of frames of second voice data.

[0045] In the example, since the first voice data is continuously obtained, the first voice data can be accumulated, and a plurality of first voice data can be used to form a second voice data of a second number of frames, and the plurality of first voice data can be arranged in chronological order to obtain the second voice data. The second number of frames is greater than the first number of frames, wherein the first number of frames and the second number of frames can be set in combination with actual application conditions.

[0046] Step S150: using a second speech recognition model to recognize a second text corresponding to the second speech data.

[0047] In the example, after obtaining the second voice data of the second frame number, the second voice data can be input into a speech recognition model for speech recognition processing. The speech recognition model here is recorded as the second speech recognition model. In the embodiment of the present application, there is no specific restriction on the structure of the second speech recognition model, and it can be built using any commonly used algorithm in the field of machine learning. Since the second speech recognition model needs to recognize the second voice data, and the second voice data is formed by multiple first voice data, the scale of the second speech recognition model is greater than that of the first speech recognition model. In order to make the obtained second text more accurate, the second speech recognition model can be a large-scale speech recognition model. Exemplarily, the second speech recognition model can be a Whisper model.

[0048] Step S160: Use the second text to update the outputted multiple first texts corresponding to the multiple first voice data.

[0049] In the example, the outputted multiple first texts may be replaced by the second texts. That is, the second texts are the final speech recognition results. The second texts are the target texts.

[0050] According to the speech processing method of the embodiment of the present application, the first text corresponding to the first speech data is recognized and output by the first speech recognition model, and then the second text corresponding to the second speech data is recognized by the second speech recognition model, and the second text is used to update the output multiple first texts corresponding to the multiple first speech data. Since the number of frames of the first speech data is small, the recognition result can be output in real time using the first speech recognition model. The second speech data contains more complete information than the first data, and the second speech recognition model can be used to improve the accuracy of the output text to improve the recognition accuracy of the speech recognition result. As a result, the speech processing method of the embodiment of the present application not only realizes the output of the intermediate streaming results of speech recognition to text, but also realizes the output of the final high-accuracy recognition results, and the use of the first speech data model makes up for the deficiency of the existing speech data model that has no streaming results, effectively improving the user experience.

[0051] According to some embodiments of the present application, the scale of the first speech recognition model is smaller than the scale of the second speech recognition model.

[0052] In the examples, the size of the model usually refers to the number of parameters or layers of the model. The number of parameters refers to the total number of connection weights and bias terms in the model. The more parameters there are, the larger the capacity of the model and the more complex the representations that can be learned. The number of layers refers to the number of hidden layers in the model, including input layers and output layers. The more layers a model has, the richer the feature levels that the model can capture. These two indicators are usually used to measure the complexity and capacity of the model. Larger models usually require more computing resources and storage space, so a trade-off needs to be made between hardware resources and efficiency. Larger models may require more training data for effective training to prevent overfitting and poor generalization performance. Therefore, when choosing a model, it is necessary to weigh the size of the model according to the specific task and available resources to obtain the best performance and results.

[0053] In the example, the first speech recognition model is used to recognize the first text corresponding to the first speech data. Since the first speech data is small, the first speech recognition model can select a small-scale speech model, and can recognize the first text corresponding to the first speech data, that is, the intermediate text result. At the same time, since the first speech recognition model is small, it can quickly generate the first text, so the generation rate of the first text is fast and can be generated in real time, but the result of the first text is not very accurate.

[0054] In the example, the second speech recognition model is used to recognize the second text corresponding to the second speech data of the second frame obtained by accumulating multiple first speech data. Since the second speech data is large, the second speech recognition model can select a larger-scale speech model to recognize the second text corresponding to the second speech data, which is the final text result. Since the second speech recognition model is large, it generates the second text at a slower rate, but the result of the second text is more accurate.

[0055] In an embodiment of the present application, the scale of the first speech recognition model is smaller than that of the second speech recognition model, wherein the small model has a fast inference speed, realizes rapid recognition and output of intermediate results, generates a streaming effect, and finally uses a large model to correct the overall result, so that the speech processing method realizes both the output of intermediate streaming results from speech recognition to text and the output of the final recognition results with high accuracy.

[0056] According to some embodiments of the present application, using a second speech recognition model to recognize a second text corresponding to the second speech data includes: based on the second speech data and the first text, using the second speech recognition model to recognize the second text corresponding to the second speech data.

[0057] In the example, when using the second speech recognition model for speech recognition, the first text (recognition result of the first speech model) can be used as a prompt word or hot word for the second speech recognition model, thereby accelerating the reasoning of the second speech recognition model and improving the overall recognition speed.

[0058] In an embodiment of the present application, based on the second voice data and the first text, a second voice recognition model is used to recognize the second text corresponding to the second voice data, which can improve the recognition efficiency of the second voice model in recognizing the second voice data, thereby improving the voice recognition rate of the second voice model.

[0059] According to some embodiments of the present application, the real-time input voice data is processed to obtain the first voice data of the first frame number, including: segmenting the real-time input voice data in chronological order to obtain the third voice data of the third frame number; obtaining the fourth voice data of the first predetermined frame number after the third voice data, and the fifth voice data of the second predetermined frame number before the third voice data; wherein the first voice data includes the third voice data, the fourth voice data and the fifth voice data.

[0060] Exemplarily, the first number of frames is 30 frames, wherein the third number of frames may be 10 frames, the first predetermined number of frames may be 10 frames, and the second predetermined number of frames may be 10 frames. After obtaining 10 frames of voice data in real time, the most recent 10 frames of voice data may be obtained from the most recently obtained voice data, and 10 frames of voice data may be accumulated to form 30 frames of voice data before processing.

[0061] In the embodiment of the present application, the first voice data may include historical data and future data, which can further improve the recognition accuracy of the voice processing method.

[0062] According to some embodiments of the present application, the first speech recognition model includes a Distill-Whisper model, and the second speech recognition model includes a Whisper model.

[0063] In the example, an optional model structure of a first speech recognition model is proposed, and the model may be a Distill-Whisper model. The Distill-Whisper model is a model that optimizes and simplifies the core functions of the Whisper model. Compared with the Whisper model, its scale is greatly reduced and its speed is greatly improved. It is suitable for use in application scenarios that require fast response, such as real-time translation, real-time meeting records, etc. The Distill-Whisper model can be used as an assistant model of the Whisper model for speculative decoding to improve the speed of speech recognition.

[0064] In the example, an optional model structure of a second speech recognition model is proposed, which can be a Whisper model. The Whisper model adopts a simple end-to-end approach, that is, the encoder-decoder structure of the Transformer model. The input audio is divided into 30-second modules, then converted into a log-Mel spectrogram, and then passed to the encoder for attention calculation. Finally, the decoder predicts the corresponding text, which has a higher accuracy than other speech recognition models.

[0065] According to another aspect of the present application, a speech recognition method is provided. Figure 2 The speech recognition method according to the embodiment of the present application is shown in FIG. Figure 1 .like Figure 2 As shown, the speech recognition method includes: step S210, real-time collection of user speech data; step S220, processing the speech data to obtain first speech data of a first frame number; step S230, using a first speech recognition model to recognize a first text corresponding to the first speech data; step S240, outputting the first text; step S250, accumulating multiple first speech data to obtain second speech data of a second frame number; step S260, using a second speech recognition model to recognize a second text corresponding to the second speech data; step S270, using the second text to update the output multiple first texts corresponding to the multiple first speech data.

[0066] In practical applications, the speech recognition method of the embodiment of the present application can be applied to a variety of scenarios, such as human-computer interaction, intelligent question and answer, voice assistant, conference, live broadcast, and other scenarios where there is a need for subtitle output. The technical solution of the embodiment of the present application can be executed by the server, outputting the first text and updating the output result with the second text, which can be that the server sends the first text to the corresponding client, the client displays the first text, and the server sends the second text to the client, instructing the client to update its display result with the second text, etc. Of course, it can also be executed by the client.

[0067] In addition, in the local all-in-one machine scenario where there is a need for voice recognition, the technology of the embodiment of the present application can also be executed by the local all-in-one machine, which can realize recognition and output processing, and can realize voice collection, recognition and real-time output of recognized text without networking. Local all-in-one machines are widely used in scenarios such as banks and courts to provide users with self-service, etc.

[0068] For example, Figure 3 The speech recognition method according to the embodiment of the present application is shown in FIG. Figure 2 .like Figure 3 As shown, step S310, real-time collection of user voice data input by the user through the microphone, the user voice data including "How is the weather in Beijing today?"; step S320, processing the voice data to obtain first voice data of a first frame number, wherein the first frame number is 1; step S330, using a first voice recognition model to recognize a first text corresponding to the first voice data, wherein the scale of the first voice recognition model is smaller than that of the second voice recognition model, and the first voice recognition model can be a Distill-Whisper model; step S340, outputting the first text, wherein since the first voice data will be continuously obtained, the first text will also be continuously output, and multiple first texts will be arranged in chronological order. The first text may be composed of single words, and the output of the first text may be output word by word; step S350, accumulating multiple first voice data to obtain second voice data of a second frame number, and the second voice data may be "How is the weather in Beijing today?"; step S360, using a second voice recognition model to recognize a second text corresponding to the second voice data, wherein the scale of the second voice recognition model is greater than that of the first voice recognition model, the second voice recognition model may be a Whisper model, and the second text is "How is the weather in Beijing today?"; step S370, using the second text to update the output multiple first texts corresponding to the multiple first voice data.

[0069] In an embodiment of the present application, a first speech recognition model is used to recognize and output the first text corresponding to the first speech data, and then a second speech recognition model is used to recognize the second text corresponding to the second speech data, and the second text is used to update the output multiple first texts corresponding to the multiple first speech data. Since the number of frames of the first speech data is small, the recognition result can be output in real time using the first speech recognition model. The second speech data contains more complete information than the first data, and the second speech recognition model can be used to improve the accuracy of the output text, so as to improve the recognition accuracy of the speech recognition result. As a result, the speech processing method of the embodiment of the present application realizes the output of the intermediate streaming results of speech recognition to text, and realizes the output of the final high-accuracy recognition results, and the use of the first speech data model makes up for the deficiency of the existing speech data model that has no streaming results, effectively improving the user experience.

[0070] According to another aspect of the present application, a speech processing device is provided. Figure 4 FIG. 4 shows a structural block diagram of a speech processing device according to an embodiment of the present application. Figure 4 As shown, the speech processing device includes: a first acquisition module 410, configured to process the real-time input speech data to obtain the first speech data of the first frame number; a first recognition module 420, configured to use the first speech recognition model to recognize the first text corresponding to the first speech data; an output module 430, configured to output the first text; a second acquisition module 440, configured to accumulate multiple first speech data to obtain the second speech data of the second frame number; a second recognition module 450, configured to use the second speech recognition model to recognize the second text corresponding to the second speech data; an update module 460, configured to use the second text to update the output multiple first texts corresponding to the multiple first speech data.

[0071] Since the first acquisition module 410, the first recognition module 420, the output module 430, the second acquisition module 440, the second recognition module 450, the update module 460 used in the speech processing device can respectively correspond to Figure 1 The steps S110, S120, S130, S140, S150 and S160 are described in detail, so the details of each aspect are not repeated here.

[0072] In some embodiments, the second recognition module 450 is further configured to recognize a second text corresponding to the second voice data using a second voice recognition model based on the second voice data and the first text.

[0073] In some embodiments, the first acquisition module 410 is also configured to segment the real-time input voice data in chronological order to obtain third voice data of a third frame number; obtain fourth voice data of a first predetermined frame number after the third voice data, and fifth voice data of a second predetermined frame number before the third voice data; wherein the first voice data includes the third voice data, the fourth voice data and the fifth voice data.

[0074] According to an embodiment of the present application, a first speech recognition model is used to recognize and output the first text corresponding to the first speech data, and then a second speech recognition model is used to recognize the second text corresponding to the second speech data, and the second text is used to update the output multiple first texts corresponding to the multiple first speech data. Since the number of frames of the first speech data is small, the recognition result can be output in real time using the first speech recognition model. The second speech data contains more complete information than the first data, and the second speech recognition model can be used to improve the accuracy of the output text, so as to improve the recognition accuracy of the speech recognition result. As a result, the speech processing method of the embodiment of the present application not only realizes the output of the intermediate streaming results of speech recognition to text, but also realizes the output of the final high-accuracy recognition results, and the use of the first speech data model makes up for the deficiency of the existing speech data model that there is no streaming result, which effectively improves the user experience.

[0075] According to another aspect of the present application, a speech recognition device is provided. Figure 5 A structural block diagram of a speech recognition device according to an embodiment of the present application is shown. Figure 5 As shown, the speech recognition device includes: a collection module 510, configured to collect user speech data in real time; a first processing module 520, configured to process the speech data to obtain a first frame number of first speech data; a first speech recognition module 530, configured to use a first speech recognition model to recognize a first text corresponding to the first speech data; a text output module 540, configured to output the first text; a second processing module 550, configured to accumulate multiple first speech data to obtain a second frame number of second speech data; a second speech recognition module 560, configured to use a second speech recognition model to recognize the second speech data and the corresponding second text; a text update module 570, configured to use the second text to update the output multiple first texts corresponding to the multiple first speech data.

[0076] Since the acquisition module 510, the first processing module 520, the first speech recognition module 530, the text output module 540, the second processing module 550, the second speech recognition module 560 and the text update module 570 used in the speech recognition device can respectively correspond to Figure 1The steps S210, S220, S230, S240, S250, S260 and S270 are described in detail, so the details of each aspect are not repeated here.

[0077] According to an embodiment of the present application, a first speech recognition model is used to recognize and output the first text corresponding to the first speech data, and then a second speech recognition model is used to recognize the second text corresponding to the second speech data, and the second text is used to update the output multiple first texts corresponding to the multiple first speech data. Since the number of frames of the first speech data is small, the recognition result can be output in real time using the first speech recognition model. The second speech data contains more complete information than the first data, and the second speech recognition model can be used to improve the accuracy of the output text, so as to improve the recognition accuracy of the speech recognition result. As a result, the speech processing method of the embodiment of the present application not only realizes the output of the intermediate streaming results of speech recognition to text, but also realizes the output of the final high-accuracy recognition results, and the use of the first speech data model makes up for the deficiency of the existing speech data model that there is no streaming result, which effectively improves the user experience.

[0078] According to another aspect of the present application, a computing device is also provided, comprising at least one processor; and at least one memory communicatively connected to the at least one processor, wherein the at least one memory stores instructions, and when the instructions are executed individually or collectively by the at least one processor, the computing device executes the speech recognition method or speech processing method of any one of the aforementioned embodiments.

[0079] Figure 6 A block diagram of a computing device according to an embodiment of the present application is shown.

[0080] For example, the above-mentioned speech processing device and speech recognition device may be fully or at least partially implemented by the computing device 600 or a similar device or system.

[0081] The computing device 600 can be a variety of different types of devices. Examples of the computing device 600 include, but are not limited to, industrial control computers, desktop computers, server computers, laptop or netbook computers, mobile devices (e.g., tablet computers, cellular or other wireless phones (e.g., smart phones), notepad computers, mobile stations), and the like.

[0082] The computing device 600 may include at least one processor 602, memory 604, communication interface(s) 606, a display device 608, other input / output (I / O) devices 610, and one or more mass storage devices 612, all capable of communicating with one another, such as via a system bus 614 or other appropriate connections.

[0083] The processor 602 may be a single processing unit or multiple processing units, all of which may include a single or multiple computing units or multiple cores. The processor 602 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Among other capabilities, the processor 602 may be configured to obtain and execute computer-readable instructions stored in the memory 604, mass storage device 612, or other computer-readable media, such as program code of an operating system 616, program code of an application program 618, program code of other programs 620, and the like.

[0084] The memory 604 and the mass storage device 612 are examples of computer-readable storage media for storing instructions that are executed by the processor 602 to implement the various functions described above. For example, the memory 604 may generally include both volatile memory and non-volatile memory (e.g., RAM, ROM, etc.). In addition, the mass storage device 612 may generally include a hard drive, a solid-state drive, a removable medium, including external and removable drives, a memory card, a flash memory, a floppy disk, an optical disk (e.g., a CD, a DVD), a storage array, a network attached storage, a storage area network, etc. The memory 604 and the mass storage device 612 may all be collectively referred to herein as memory or computer-readable storage media, and may be a non-transitory medium capable of storing computer-readable, processor-executable program instructions as computer program code, which may be executed by the processor 602 as a specific machine configured to implement the operations and functions described in the examples herein.

[0085] A number of programs may be stored on mass storage device 612. These programs include operating system 616, one or more application programs 618, other programs 620, and program data 622, and they may be loaded into memory 604 for execution. Examples of such applications or program modules may include, for example, computer program logic (e.g., computer program code or instructions) for implementing the embodiments described herein.

[0086] Although in Figure 6 604 of computing device 600, but operating system 616, application programs 618, other programs 620, and program data 622, or portions thereof, may be implemented using any form of computer-readable media accessible by computing device 600. As used herein, "computer-readable media" includes at least two types of computer-readable media, namely, computer-readable storage media and communication media.

[0087] Computer-readable storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information, such as computer-readable instructions, data structures, program modules, or other data. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD), or other optical storage device, magnetic cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or any other non-transmission medium that can be used to store information for access by a computing device. In contrast, communication media can embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transmission mechanism. Computer-readable storage media as defined herein do not include communication media.

[0088] One or more communication interfaces 606 are used to exchange data with other devices, such as through a network, a direct connection, etc. Such communication interfaces can be one or more of the following: any type of network interface (e.g., a network interface card (NIC)), a wired or wireless (such as IEEE 802.11 wireless LAN (WLAN)) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a cellular network ... TM The communication interface 606 may facilitate communication within a variety of network and protocol types, including wired networks (e.g., LAN, cable, etc.) and wireless networks (e.g., WLAN, cellular, satellite, etc.), the Internet, etc. The communication interface 606 may also provide for communication with external storage devices (not shown) such as in a storage array, network attached storage, storage area network, etc.

[0089] In some examples, a display device 608 such as a monitor may be included for displaying information and images to the user. Other I / O devices 610 may be devices that receive various inputs from the user and provide various outputs to the user, and may include a touch input device, a gesture input device, a camera, a keyboard, a remote control, a mouse, a printer, an audio input / output device, and the like.

[0090] The technology described herein can be supported by these various configurations of the computing device 600, and is not limited to the specific examples of the technology described herein. For example, the functionality can also be implemented in whole or in part on the "cloud" by using a distributed system. The cloud includes and / or represents a platform for resources. The platform abstracts the underlying functionality of the hardware (e.g., server) and software resources of the cloud. Resources can include applications and / or data that can be used when performing computing processing on a server away from the computing device 600. Resources can also include services provided over the Internet and / or through a subscriber network such as a cellular or Wi-Fi network. The platform can abstract resources and functions to connect the computing device 600 to other computing devices. Therefore, the implementation of the functions described herein can be distributed throughout the cloud. For example, the functions can be implemented partially on the computing device 600 and partially through a platform that abstracts the functions of the cloud.

[0091] According to another aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the speech recognition method or speech processing method of any one of the aforementioned embodiments is implemented during execution.

[0092] According to another aspect of the present application, a computer program product is also provided, comprising instructions, which, when executed individually or collectively by one or more processors of a computing device, cause the computing device to execute the speech recognition method or speech processing method of any one of the aforementioned embodiments.

[0093] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application, and they should all be included in the scope of the claims and specification of the present application. In particular, as long as there is no structural conflict, the various technical features mentioned in the various embodiments can be combined in any way. The present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions that fall within the scope of the claims.

Claims

1. A speech processing method, characterized in that: include: Processing the voice data input in real time to obtain first voice data of a first number of frames; Recognize a first text corresponding to the first voice data using a first voice recognition model; outputting the first text; Accumulate a plurality of the first voice data to obtain a second number of frames of second voice data; Recognize a second text corresponding to the second voice data using a second voice recognition model; The outputted plurality of first texts corresponding to the plurality of first voice data are updated using the second text.

2. The method according to claim 1, characterized in that The scale of the first speech recognition model is smaller than the scale of the second speech recognition model.

3. The method according to claim 1 or 2, characterized in that: The step of using a second speech recognition model to recognize a second text corresponding to the second speech data includes: Based on the second voice data and the first text, a second text corresponding to the second voice data is recognized using a second voice recognition model.

4. The method according to claim 1, characterized in that The step of processing the real-time input voice data to obtain the first voice data of the first number of frames includes: Segmenting the real-time input voice data in chronological order to obtain third voice data of a third number of frames; Acquire fourth voice data of a first predetermined number of frames after the third voice data, and fifth voice data of a second predetermined number of frames before the third voice data; The first voice data includes the third voice data, the fourth voice data and the fifth voice data.

5. The method according to claim 1 or 2, characterized in that: The first speech recognition model includes a Distill-Whisper model, and the second speech recognition model includes a Whisper model.

6. A speech recognition method, characterized in that: include: Collect user voice data in real time; Processing the voice data to obtain first voice data of a first number of frames; Recognize a first text corresponding to the first voice data using a first voice recognition model; outputting the first text; Accumulate a plurality of the first voice data to obtain a second number of frames of second voice data; Recognize a second text corresponding to the second voice data using a second voice recognition model; The outputted plurality of first texts corresponding to the plurality of first voice data are updated using the second text.

7. A speech processing device, characterized in that: include: A first acquisition module is configured to process the real-time input voice data to obtain first voice data of a first number of frames; A first recognition module is configured to recognize a first text corresponding to the first voice data using a first voice recognition model; An output module, configured to output the first text; A second acquisition module is configured to accumulate a plurality of the first voice data to obtain a second number of frames of second voice data; A second recognition module is configured to recognize a second text corresponding to the second voice data using a second voice recognition model; An updating module is configured to update the outputted multiple first texts corresponding to the multiple first voice data using the second text.

8. A speech recognition device, characterized in that: include: A collection module, configured to collect user voice data in real time; A first processing module is configured to process the voice data to obtain first voice data of a first number of frames; A first speech recognition module is configured to recognize a first text corresponding to the first speech data using a first speech recognition model; A text output module, configured to output the first text; A second processing module is configured to accumulate a plurality of the first voice data to obtain a second number of frames of second voice data; A second speech recognition module is configured to recognize the second speech data and the corresponding second text using a second speech recognition model; The text updating module is configured to update the outputted multiple first texts corresponding to the multiple first voice data using the second text.

9. A computing device, characterized in that include: at least one processor; as well as At least one memory communicatively connected to the at least one processor, the at least one memory storing instructions, which, when executed individually or collectively by the at least one processor, enable the computing device to execute the speech processing method described in any one of claims 1 to 5 or the speech recognition method described in claim 6.

10. A computer-readable storage medium, characterized in that: Instructions are stored, which, when executed individually or collectively by one or more processors of a computing device, cause the computing device to execute the speech processing method according to any one of claims 1 to 5 or the speech recognition method according to claim 6.

11. A computer program product, characterized in that The method comprises instructions which, when executed individually or collectively by one or more processors of a computing device, cause the computing device to execute the speech processing method according to any one of claims 1 to 5 or the speech recognition method according to claim 6.

Images