A speech recognition method, apparatus and electronic device

Abstract

The present disclosure relates to the technical field of human-computer interaction, and particularly relates to a speech recognition method and device and electronic equipment, and is used for solving the problem of how to improve the accuracy of speech recognition of the electronic equipment. The method comprises the following steps: obtaining a fusion score of a candidate recognition result of each to-be-recognized audio data based on five aspects of actual score of the to-be-recognized audio data, actual length of the to-be-recognized audio data, predicted length of the to-be-recognized audio data, theoretical length of the to-be-recognized audio data and theoretical score of the to-be-recognized audio data; determining an actual recognition result of the candidate recognition result of the to-be-recognized audio data based on the fusion score of the to-be-recognized audio data.

Description

Technical Field

[0001] This disclosure relates to the field of human-computer interaction technology, and in particular to a speech recognition method, apparatus and electronic device. Background Technology

[0002] In human-computer interaction scenarios, speech recognition plays a crucial role in natural language understanding and generation. The accuracy of speech recognition is both the foundation and the bottleneck for downstream tasks. For example, when using electronic devices, users can control them via voice. During this process, the electronic device continuously receives the audio stream corresponding to the user's voice input, decodes it, and provides real-time feedback on the decoding results. When the speech recognition accuracy of an electronic device is low, the user needs to repeatedly input the same voice for the device to correctly recognize the corresponding action, resulting in a poor user experience.

[0003] Therefore, improving the accuracy of voice recognition in electronic devices has become an urgent problem to be solved. Summary of the Invention

[0004] This disclosure provides a speech recognition method, apparatus, and electronic device.

[0005] The technical solution disclosed herein is as follows:

[0006] In a first aspect, this disclosure provides a server, comprising: a communicator configured to acquire audio data to be recognized; a processor configured to recognize audio features of the audio data to be recognized acquired by the communicator, obtaining at least one acoustic feature corresponding to the audio data to be recognized, and an actual score corresponding to each acoustic feature; the processor is further configured to predict the acoustic features, determining at least one candidate recognition result containing the acoustic features, and the actual length of the text contained in each candidate recognition result; the processor is further configured to predict the number of characters in the audio data to be recognized acquired by the communicator, obtaining the predicted length of the text contained in the audio data to be recognized; the processor is further configured to perform speech activity detection on the audio data to be recognized acquired by the communicator, obtaining the theoretical length of the text contained in the audio data to be recognized; the processor is further configured to score the candidate recognition results, obtaining a theoretical score corresponding to each candidate recognition result; the processor is further configured to obtain a fusion score for each candidate recognition result based on the actual score, actual length, predicted length, theoretical length, and theoretical score; and the processor is further configured to determine the actual recognition result of the audio data to be recognized based on the fusion score.

[0007] Secondly, this disclosure provides a speech recognition method, comprising: acquiring audio data to be recognized; recognizing audio features of the audio data to be recognized to obtain at least one acoustic feature corresponding to the audio data to be recognized, and an actual score corresponding to each acoustic feature; predicting the acoustic features to determine at least one candidate recognition result containing the acoustic features, and the actual length of the text contained in each candidate recognition result; predicting the number of characters in the audio data to be recognized to obtain the predicted length of the text contained in the audio data to be recognized; performing speech activity detection on the audio data to be recognized to obtain the theoretical length of the text contained in the audio data to be recognized; scoring the candidate recognition results to obtain a theoretical score corresponding to each candidate recognition result; obtaining a fusion score for each candidate recognition result based on the actual score, actual length, predicted length, theoretical length, and theoretical score; and determining the actual recognition result of the audio data to be recognized based on the fusion score.

[0008] Thirdly, this disclosure provides an electronic device, including: a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to cause the electronic device to implement the speech recognition method as provided in any of the second aspects when executing the computer program.

[0009] Fourthly, the present invention provides a computer-readable storage medium, comprising: storing a computer program on the computer-readable storage medium, the computer program being executed by a processor of a speech recognition method as provided in any of the second aspects.

[0010] Fifthly, the present invention provides a computer program product that, when run on a computer, causes the computer to perform a speech recognition method as provided in any of the second aspects.

[0011] It should be noted that the aforementioned computer instructions may be stored, in whole or in part, on the first computer-readable storage medium. The first computer-readable storage medium may be packaged together with the server's processor or may be packaged separately from the server's processor; this disclosure does not impose any limitations on this.

[0012] The descriptions of the second, third, fourth, and fifth aspects in this disclosure can be referenced to the detailed description of the first aspect; and the beneficial effects of the descriptions of the second, third, fourth, and fifth aspects can be referenced to the analysis of the beneficial effects of the first aspect, which will not be repeated here.

[0013] In this disclosure, the names of the aforementioned servers do not limit the devices or functional modules themselves. In actual implementation, these devices or functional modules may appear under other names. As long as the functions of each device or functional module are similar to those of this disclosure, they fall within the scope of the claims of this disclosure and their equivalents.

[0014] These or other aspects of this disclosure will become more readily apparent in the following description.

[0015] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0016] As can be seen, in the speech recognition method provided in this disclosure, after the server obtains the audio data to be recognized, it identifies the audio features of the audio data to be recognized to obtain at least one acoustic feature corresponding to the audio data to be recognized, and the actual score corresponding to each acoustic feature. Then, the server predicts the acoustic features to determine at least one candidate recognition result containing the acoustic features, and the actual length of the text contained in each candidate recognition result. To ensure that the actual recognition result contains the correct text length, the speech recognition method provided in this disclosure involves the server predicting the predicted length of the text contained in the audio data to be recognized by performing text word count prediction; the server performing speech activity detection on the audio data to be recognized to obtain the theoretical length of the text contained in the audio data to be recognized; the server scoring the candidate recognition results to obtain the theoretical score corresponding to each candidate recognition result; and then, the server obtaining a fusion score for each candidate recognition result based on the actual score, actual length, predicted length, theoretical length, and theoretical score. In this way, the predicted length and theoretical length can be used as prediction benchmarks to quantify the theoretical score of the candidate recognition results, so that the text length of the actual recognition result of the audio data to be recognized is biased towards the prediction benchmark, thereby reducing the phenomenon of extra words and missing words in the translation results, improving the accuracy of the actual recognition results, and solving the problem of how to improve the accuracy of speech recognition of electronic devices. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0018] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A schematic diagram illustrating a scenario for the speech recognition method provided in an embodiment of this application;

[0020] Figure 2 This is one of the structural schematic diagrams of the display device provided in the embodiments of this application;

[0021] Figure 3 This is a second schematic diagram of the structure of the display device provided in the embodiments of this application;

[0022] Figure 4 One of the flowcharts of the speech recognition method provided in the embodiments of this application;

[0023] Figure 5 A schematic diagram of the network structure of the audio text word count prediction model for the speech recognition method provided in the embodiments of this application;

[0024] Figure 6 A schematic diagram illustrating the speech activity detection of the audio data to be recognized in the speech recognition method provided in this application embodiment;

[0025] Figure 7 A second schematic flowchart illustrating the speech recognition method provided in this application embodiment;

[0026] Figure 8 The third schematic flowchart of the speech recognition method provided in the embodiments of this application;

[0027] Figure 9 The fourth flowchart illustrating the speech recognition method provided in this application embodiment;

[0028] Figure 10 Fifth schematic flowchart of the speech recognition method provided in the embodiments of this application;

[0029] Figure 11 A flowchart illustrating the speech recognition method provided in this application embodiment is shown in Figure 6.

[0030] Figure 12 Seventh schematic flowchart of the speech recognition method provided in the embodiments of this application;

[0031] Figure 13 This is a schematic diagram of the server structure provided in an embodiment of this application;

[0032] Figure 14 This is a schematic diagram of a chip system provided in an embodiment of this application. Detailed Implementation

[0033] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0034] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0035] The display device provided in this application can have various implementation forms, such as a television, a smart television, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc. Figure 1 and Figure 2 This is one specific embodiment of the display device of this application.

[0036] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0037] The acoustic model in this embodiment is one of the most important parts of the speech recognition system, and most mainstream systems use Hidden Markov Models for modeling.

[0038] In this embodiment of the disclosure, the Dropout layer refers to a layer in a neural network in which some neurons have been removed.

[0039] The language model (LM) in this embodiment plays an important role in natural language processing. Its task is to predict the probability of a sentence appearing in a language.

[0040] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device according to one or more embodiments of this application, such as... Figure 1 As shown, a user can operate the display device 200 via a mobile terminal 300 and a control device 100. The control device 100 can be a remote control, and communication between the remote control and the display device includes infrared protocol communication, Bluetooth protocol communication, wireless or other wired methods to control the display device 200. The user can input user commands through buttons on the remote control, voice input, control panel input, etc., to control the display device 200. In some embodiments, a mobile terminal, tablet computer, computer, laptop computer, and other smart devices can also be used to control the display device 200.

[0041] In some embodiments, the electronic device provided in this application can be the aforementioned display device 200. When a user needs to perform a voice search while using the display device 200, they can press the voice button on the control device 100 and input the voice message to search for information, or input the voice message through another electronic device (such as a mobile phone) that has established a communication connection with the display device 200. Upon receiving the voice message, the display device 200 performs preprocessing on the voice message (e.g., quantization, sampling, pre-filtering, pre-emphasis, and frame windowing of the audio signal, to remove noise from the search information), thus obtaining the audio stream to be identified. Next, the display device 200 identifies the audio features of the audio data to be recognized, obtaining at least one acoustic feature corresponding to the audio data to be recognized, and the actual score corresponding to each acoustic feature; the display device 200 predicts the acoustic features, determining at least one candidate recognition result containing the acoustic features, and the actual length of the text contained in each candidate recognition result; the display device 200 predicts the number of characters in the audio data to be recognized, obtaining the predicted length of the text contained in the audio data to be recognized; the display device 200 performs speech activity detection on the audio data to be recognized, obtaining the theoretical length of the text contained in the audio data to be recognized; the display device 200 scores the candidate recognition results, obtaining the theoretical score corresponding to each candidate recognition result; based on the actual score, actual length, predicted length, theoretical length, and theoretical score, the display device 200 obtains a fusion score for each candidate recognition result; based on the fusion score, the display device 200 determines the actual recognition result of the audio data to be recognized. Then, the display device 200 displays the actual recognition result.

[0042] In some embodiments, the electronic device provided in this application can be the server 400 described above. When a user needs to perform a voice search while using the display device 200, they can press the voice button on the control device 100 and input the voice message to search for information, or input the voice message through another electronic device (such as a mobile phone) that has established a communication connection with the display device 200. Upon receiving the voice message, the display device 200 sends it to the server 400. The server 400 receives the voice message sent by the display device 200. Then, the server 400 performs preprocessing on the voice message (e.g., quantization, sampling, pre-filtering, pre-emphasis, and frame windowing of the audio signal, to remove noise from the search information), thus obtaining the audio stream to be identified. Next, server 400 identifies the audio features of the audio data to be recognized, obtaining at least one acoustic feature corresponding to the audio data to be recognized, and the actual score corresponding to each acoustic feature; server 400 predicts the acoustic features, determining at least one candidate recognition result containing the acoustic features, and the actual length of the text contained in each candidate recognition result; server 400 predicts the number of characters in the audio data to be recognized, obtaining the predicted length of the text contained in the audio data to be recognized; server 400 performs speech activity detection on the audio data to be recognized, obtaining the theoretical length of the text contained in the audio data to be recognized; server 400 scores the candidate recognition results, obtaining the theoretical score corresponding to each candidate recognition result; server 400 obtains a fusion score for each candidate recognition result based on the actual score, actual length, predicted length, theoretical length, and theoretical score; server 400 determines the actual recognition result of the audio data to be recognized based on the fusion score. Then, server 400 can send the actual recognition result to display device 200 so that display device 200 displays the actual recognition result.

[0043] As can be seen, the speech recognition method provided in this disclosure predicts the length of the text contained in the audio data to be recognized by predicting the number of characters in the audio data. Simultaneously, energy-based voice activity detection (VAD) also yields the theoretical length of the text contained in the audio data. Then, using the predicted length and theoretical length as prediction benchmarks, the theoretical score of the candidate recognition result is quantified, ensuring that the actual recognition result of the audio data is closer to the predicted benchmark in terms of text length. This reduces the occurrence of extra or missing characters in the translation results, thus guaranteeing the accuracy of the actual recognition result.

[0044] Figure 2 A hardware configuration block diagram of a display device 200 according to an exemplary embodiment is shown. For example... Figure 2The display device 200 shown includes at least one of the following: a tuner / demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface 280. The controller includes a central processing unit, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first to nth interface for input / output. The display 260 may be a touch-enabled display, such as a touch screen display. The tuner / demodulator 210 receives broadcast television signals via wired or wireless reception and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals. The detector 230 is used to collect signals from the external environment or signals interacting with the external environment. The controller 250 and the tuner / demodulator 210 may be located in different separate devices; that is, the tuner / demodulator 210 may also be located in an external device of the main device containing the controller 250, such as an external set-top box.

[0045] In some embodiments, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200.

[0046] In some examples, the display device 200 of one or more embodiments is a television set 1, and the operating system of the television set 1 is the Android system, for example... Figure 3 As shown, TV 1 can be logically divided into an application layer (referred to as "application layer") 21, an application framework layer (referred to as "framework layer") 22, an Android runtime and system library layer (referred to as "system runtime library layer") 23, and a kernel layer 24.

[0047] The application layer 21 includes one or more applications. These applications can be system applications or third-party applications. For example, application layer 21 may include a first application that provides speech recognition functionality. The framework layer 22 provides application programming interfaces (APIs) and programming frameworks for the applications in application layer 21. The system runtime library layer 23 provides support for the upper layer, namely the framework layer 22. When the framework layer 22 is used, the Android operating system runs the C / C++ libraries contained in the system runtime library layer 23 to implement the functions required by the framework layer 22. The kernel layer 24 acts as software middleware between the hardware layer and application layer 21, managing and controlling hardware and software resources.

[0048] In some examples, after the TV 1 is turned on, the first application starts. When a user needs to perform a voice search while using the TV 1, they can press the voice button on the control device 100 and input the voice message they want to search for, or input the voice message through another electronic device (such as a mobile phone) that has established a communication connection with the TV 1. Upon receiving the voice message, the TV 1 sends it to the server 400. The server 400's acquisition unit 201 acquires the voice message sent by the display device 200. Then, the server 400's processing unit 202 preprocesses the voice message acquired by the acquisition unit 201 (e.g., one or more of the following: audio signal quantization, sampling, pre-filtering, pre-emphasis, and frame windowing to remove noise from the search information), thus obtaining the audio stream to be identified. Subsequently, the processing unit 202 of server 400 identifies the audio features of the audio data to be identified acquired by the acquisition unit 201, obtaining at least one acoustic feature corresponding to the audio data to be identified, and the actual score corresponding to each acoustic feature; the processing unit 202 of server 400 predicts the acoustic features, determining at least one candidate recognition result containing the acoustic features, and the actual length of the text contained in each candidate recognition result; the processing unit 202 of server 400 predicts the number of characters in the audio data to be identified, obtaining the predicted length of the text contained in the audio data to be identified; the processing unit 202 of server 400 performs speech activity detection on the audio data to be identified, obtaining the theoretical length of the text contained in the audio data to be identified; the processing unit 202 of server 400 scores the candidate recognition results, obtaining the theoretical score corresponding to each candidate recognition result; the processing unit 202 of server 400 obtains the fusion score of each candidate recognition result based on the actual score, actual length, predicted length, theoretical length, and theoretical score; the processing unit 202 of server 400 determines the actual recognition result of the audio data to be identified based on the fusion score. Then, the processing unit 202 of the server 400 controls the sending unit 203 to send the actual recognition result to the television 1 so that the television 1 can display the actual recognition result.

[0049] Specifically, when the execution subject of the speech recognition method provided in this embodiment is a server 400, the storage unit 204 of the server 400 is used to store data such as the program code of the speech recognition method provided in this embodiment.

[0050] In some examples, after the TV 1 is turned on, the first application starts. When a user needs to perform a voice search while using the TV 1, they can press the voice button on the control device 100 and input the voice of the information to be searched, or input the voice of the information to be searched through other electronic devices (such as mobile phones) that have established a communication connection with the TV 1. At this time, after the acquisition unit 101 of the TV 1 acquires the voice of the information to be searched, the processing unit 102 of the TV 1 performs preprocessing on the voice acquired by the acquisition unit 201 (such as one or more of the following: audio signal quantization, sampling, pre-filtering, pre-emphasis, and frame windowing to remove noise in the search information), thus obtaining the audio stream to be identified. Subsequently, the processing unit 102 of the television set 1 identifies the audio features of the audio data to be identified acquired by the acquisition unit 101, obtaining at least one acoustic feature corresponding to the audio data to be identified, and the actual score corresponding to each acoustic feature; the processing unit 102 of the television set 1 predicts the acoustic features, determining at least one candidate recognition result containing the acoustic features, and the actual length of the text contained in each candidate recognition result; the processing unit 102 of the television set 1 predicts the number of characters in the audio data to be identified, obtaining the predicted length of the text contained in the audio data to be identified; the processing unit 102 of the television set 1 performs speech activity detection on the audio data to be identified, obtaining the theoretical length of the text contained in the audio data to be identified; the processing unit 102 of the television set 1 scores the candidate recognition results, obtaining the theoretical score corresponding to each candidate recognition result; the processing unit 102 of the television set 1 obtains the fusion score of each candidate recognition result based on the actual score, actual length, predicted length, theoretical length, and theoretical score; the processing unit 102 of the television set 1 determines the actual recognition result of the audio data to be identified based on the fusion score. Then, the processing unit 102 of the television set 1 can control the display unit 103 to display the actual recognition result.

[0051] Specifically, when the executing entity of the speech recognition method provided in this embodiment is a television set 1, the storage unit 104 of the television set 1 is used to store data such as the program code of the speech recognition method provided in this embodiment.

[0052] In the following embodiments, the execution subject of the speech recognition method provided in this disclosure is the aforementioned server 400, and the electronic device is the aforementioned television set 1, to illustrate the method of this application embodiment.

[0053] This application provides a speech recognition method applied to a server 400, such as... Figure 4 As shown, the speech recognition method may include S11-S18.

[0054] S11. Obtain the audio data to be recognized.

[0055] In some examples, when display device 200 receives voice input from a user requesting information, it divides the voice into audio segments and uploads each segment as a buffer packet to server 400 over the network. Server 400 then continuously receives these buffer packets from display device 200, thus forming the audio stream to be recognized.

[0056] S12. Identify the audio features of the audio data to be identified, and obtain at least one acoustic feature corresponding to the audio data to be identified, and the actual score corresponding to each acoustic feature.

[0057] In some examples, the buffer packet can be fed into a feature extractor to extract audio features (such as Mel Frequency Cepstrum Coefficients (MFCCs)) to obtain at least one audio feature. The audio feature is then stored in a feature queue, awaiting decoding.

[0058] In some examples, the training process for the feature extractor is as follows:

[0059] Obtain training sample data and the labeling results of the training sample data. The training sample data includes historical recognized audio streams, and the labeling results include the audio features corresponding to the historical recognized audio streams.

[0060] The training sample data is input into the neural network model for learning, and the prediction results of the neural network model on the training sample data are obtained.

[0061] Based on the prediction and labeling results, the network parameters of the neural network model are adjusted until the neural network model converges, thus obtaining the feature extractor.

[0062] In some examples, when identifying the audio features of the audio stream to be identified, audio features can be extracted first to obtain the corresponding audio features (such as Mel Frequency Cepstrum Coefficients (MFCCs) and filter bank (Fbank)). Then, the audio features are identified to obtain the acoustic features corresponding to the audio stream to be identified, and the actual scores corresponding to the acoustic features.

[0063] In some examples, audio features can be input into a pre-trained speech model for recognition to obtain the acoustic features corresponding to the audio stream to be recognized, as well as the actual scores corresponding to the acoustic features.

[0064] In some examples, the training process for the pre-trained acoustic model is as follows:

[0065] Obtain the training sample data and its labeling results. The training sample data includes historical audio streams and their corresponding audio features. The labeling results include the acoustic features corresponding to the audio features of the historical audio streams and their actual scores.

[0066] The training sample data is input into the neural network model for learning, and the prediction results of the neural network model on the training sample data are obtained.

[0067] Based on the prediction and labeling results, the network parameters of the neural network model are adjusted until the neural network model converges, resulting in a pre-trained acoustic model.

[0068] S13. Predict the acoustic features, determine at least one candidate recognition result containing the acoustic features, and the actual length of the text contained in each candidate recognition result.

[0069] In some examples, when making predictions about acoustic features, the acoustic features can be fed into a pre-trained language model to obtain at least one candidate recognition result that includes the acoustic features.

[0070] In some examples, a pre-trained language model can predict the probability of the target text corresponding to a given acoustic feature appearing in a language (such as a word or sentence) containing that target text. Then, based on this probability, languages ​​meeting certain selection criteria are chosen as candidate recognition results. These selection criteria include a probability greater than a probability threshold (e.g., 95%), or sorting the probabilities in descending order and selecting the top N languages ​​in descending order.

[0071] In some examples, after obtaining the candidate recognition results, the actual length of the text contained in the candidate recognition results is obtained by counting the number of characters contained in the candidate recognition results.

[0072] S14. Predict the number of characters in the audio data to be recognized to obtain the predicted length of the text contained in the audio data to be recognized.

[0073] In some examples, the audio data to be identified can be input into an audio-text word count prediction model to predict the word count of the text and obtain the predicted length of the text contained in the audio data to be identified; wherein, the audio-text word count prediction model consists of an object detection algorithm model and a multi-classification model.

[0074] For example, the network structure of an audio text word count prediction model is as follows: Figure 5As shown, the model includes Spectrum Augmentation (SpecAug), Convolution Subsampling (ConvolutionSubsampling), Linear layers, Dropout layers, N Conformer Blocks (ConformerBlocks×N), and a multi-class classification model. Here, ConformerBlocks×N represents N ConformerBlock structures. Each ConformerBlock structure includes a Feed Forward Module, a Multi-Head Self Attention Module, a Convolution Module, a Feed Forward Module, and a Layer Normalization Module. Each Module is preceded by a Layer Normalization Module and followed by a Dropout layer, and all have residual chains connecting them, with the residual data being the input data itself.

[0075] In some examples, to better process the audio data to be recognized, the speech recognition method provided in this disclosure uses a 25ms window length and a 10ms window shift to extract 80-dimensional fbank features from the audio to be sampled, such as... Figure 5 As shown, the framed audio is first subjected to spectral enhancement (SpacAAugment), then downsampled by 4x convolution, and then processed sequentially through a linear layer, a dropout layer, and a multi-classification model.

[0076] When the audio data to be recognized is input into the audio-text word count prediction model for word count prediction, the acoustic model has already determined the end point of the audio data to be recognized. Therefore, the audio-text word count prediction model can predict the word count of the audio data to be recognized, thus ensuring that the audio data to be recognized input into the audio-text word count prediction model is a complete sentence.

[0077] Specifically, audio features can be extracted from the audio data to be recognized to obtain Fbank features. These Fbank features are then input into an audio-text word count prediction model for encoding, resulting in an encoded X (e.g., 256)-dimensional hidden layer vector. This hidden layer vector is then input into a multi-classification model to predict the text word count, yielding the predicted word count.

[0078] S15. Perform speech activity detection on the audio data to be recognized to obtain the theoretical length of the text contained in the audio data to be recognized.

[0079] In some examples, VAD is part of the front-end signal processing. When sound appears in an audio frame, the probability value corresponding to that audio frame is greater than a given threshold, and the label of that audio frame is set to 1; conversely, if no sound appears in an audio frame, the probability value corresponding to that audio frame is less than or equal to the given threshold, and the label of that audio frame is set to 0. Based on this principle, theoretically, the correspondence between each character in the audio data to be recognized and the audio frame can be obtained. For example, when using Automatic Speech Recognition (ASR) to recognize the audio data to be recognized, statistics begin after the characters are transcribed, counting the label 0 output by VAD, and aligning the positions according to the characters transcribed by ASR.

[0080] For example, server 400 performs speech-to-text conversion on the audio data to be recognized, obtaining each character contained in the audio data. Server 400 aligns the characters based on the tag values ​​to obtain aligned audio frames. Server 400 counts the audio frames with tag values ​​of the target value in the aligned audio frames to obtain the theoretical length of the text contained in the audio data to be recognized. For example, a target value of 1 indicates that there is sound in the audio frame, and a target value of 0 indicates that there is no sound in the audio frame. Therefore, the number of consecutive 0s and consecutive 1s can be used to determine the number of characters that the information may contain.

[0081] In this process, since VAD (Voice over Audio) may misjudge, the speech recognition method provided in this embodiment employs a smoothing technique to correct audio frames. When sound appears in an audio frame, sound also appears in several adjacent audio frames. If sound appears in the current audio frame, but no sound appears in the two adjacent audio frames, it indicates that the audio frame may be noise, and therefore it is smoothed out and ignored. Specifically, the label values ​​of audio frames that meet the smoothing condition in the aligned audio frames are set to 0 to obtain the corrected audio frames. The smoothing condition includes that the label value of the current audio frame is different from the label values ​​of the two adjacent audio frames, and the label values ​​of the two adjacent audio frames are 0. The audio frames with label values ​​of the target value in the corrected audio frames are statistically analyzed to obtain the theoretical length of the text contained in the audio data to be recognized. A label value of 0 indicates that no sound appears in the audio frame. Figure 6 As shown, by using smoothing techniques to correct the aligned audio frames, the audio frame corresponding to each character can be determined more accurately.

[0082] S16. Score the candidate recognition results to obtain the theoretical score corresponding to each candidate recognition result.

[0083] In some examples, the candidate recognition results are input into a Chinese language model (N-Gram) for scoring, yielding a theoretical score for each candidate recognition result. The Chinese language model is trained using the recognition results and their labeled scores.

[0084] The training process for the Chinese language model is as follows:

[0085] Obtain training sample data and the labeling results of the training sample data. The training sample data includes at least one training recognition result, and the labeling results include the label score corresponding to the training recognition result.

[0086] The training sample data is input into the neural network model for learning, and the prediction results of the neural network model on the training sample data are obtained.

[0087] Based on the prediction and labeling results, the network parameters of the neural network model are adjusted until the neural network model converges, thus obtaining the Chinese language model.

[0088] S17. Based on the actual score, actual length, predicted length, theoretical length, and theoretical score, obtain the fusion score for each candidate recognition result.

[0089] In some examples, the first weight value of the candidate recognition result corresponding to the actual length in the fusion score can be determined based on the actual length, predicted length, and theoretical length. The fusion score of each candidate recognition result is obtained based on the second weight value of the acoustic features contained in the candidate recognition result in the fusion score, the actual score, the first weight value, and the theoretical score. The fusion score satisfies the following formula:

[0090] P(w i |o)=α×γ i +ρ×β i ×lm i ;

[0091]

[0092] Where, o represents acoustic feature, w i Let i represent the candidate identification result, P(w). i |o) represents the fusion score of the candidate recognition results corresponding to the acoustic features, α represents the second weight value, and γ i ρ represents the sum of the actual scores corresponding to each acoustic feature included in candidate recognition result i, ρ represents the third weight value of candidate recognition result i in the fusion score, and β represents the third weight value of candidate recognition result i in the fusion score. i lm represents the first weight value corresponding to candidate recognition result i. i Let N represent the theoretical score of candidate identification result i, N represent the prediction length, M represent the theoretical length, and n represent the theoretical score of candidate identification result i. iLet represent the actual length of candidate recognition result i, i∈[1,X], where X is the total number of candidate recognition results, i and X are both integers, and α and ρ are both constants greater than 0.

[0093] S18. Based on the fusion score, the actual recognition result is determined.

[0094] In some examples, the actual recognition result of the audio data to be recognized can be determined based on the fusion score, which is the candidate recognition result corresponding to the maximum value of the fusion score.

[0095] In some examples, fusion scores can be used to filter candidates whose scores are greater than or equal to a score threshold (e.g., 90 points). If only one fusion score is greater than or equal to the score threshold, the candidate recognition result corresponding to that score is selected as the final candidate recognition result. If multiple fusion scores are greater than or equal to the score threshold, the candidate recognition result corresponding to the highest fusion score is selected. Alternatively, if multiple fusion scores are greater than or equal to the score threshold, a candidate recognition result corresponding to a randomly selected fusion score is chosen as the final candidate recognition result.

[0096] As described above, the speech recognition method provided in this disclosure predicts the length of the text contained in the audio data to be recognized by predicting the number of characters in the audio data. Simultaneously, energy-based VAD (Voice Aspect Difference) also yields the theoretical length of the text contained in the audio data. Then, using the predicted length and theoretical length as prediction benchmarks, the theoretical score of the candidate recognition result is quantified, ensuring that the actual recognition result of the audio data is closer to the prediction benchmark in terms of text length. This reduces the occurrence of extra or missing characters in the translation result, thus guaranteeing the accuracy of the actual recognition result.

[0097] In some feasible examples, combining Figure 4 ,like Figure 7 The above S14 can be implemented in the following S140.

[0098] S140. Input the audio data to be identified into the audio-text word count prediction model to predict the word count of the text and obtain the predicted length of the text contained in the audio data to be identified. The audio-text word count prediction model consists of an object detection algorithm model and a multi-classification model.

[0099] In some feasible examples, combining Figure 4 ,like Figure 8 The above S15 can be implemented through the following S150-S153.

[0100] S150. Perform speech activity detection on the audio data to be recognized and determine the label value corresponding to each audio frame in the audio data to be recognized.

[0101] S151. Perform speech-to-text conversion on the audio data to be recognized to obtain each character contained in the audio data to be recognized.

[0102] S152. Align the audio frames based on the tag values ​​and characters to obtain the aligned audio frames.

[0103] S153. Statistically analyze the audio frames with label values ​​of the target value in the aligned audio frames to obtain the theoretical length of the text contained in the audio data to be identified. The text contains at least one character, and each character corresponds to one or more audio frames with label values ​​of the target value.

[0104] In some feasible examples, combining Figure 8 ,like Figure 9 The above S153 can be specifically implemented through the following S1530 and S1531.

[0105] S1530. Set the label values ​​of audio frames that meet the smoothing conditions in the aligned audio frames to 0 to obtain the corrected audio frames. The smoothing conditions include that the label value of the current audio frame is different from the label values ​​of the two adjacent audio frames, and the label values ​​of the two adjacent audio frames are 0.

[0106] S1531. Statistically analyze the audio frames with label values ​​of the target value in the corrected audio frames to obtain the theoretical length of the text contained in the audio data to be identified.

[0107] In some feasible examples, combining Figure 4 ,like Figure 10 The above S16 can be implemented in the following S160.

[0108] S160. Input the candidate recognition results into the Chinese language model for scoring, and obtain the theoretical score corresponding to each candidate recognition result. The Chinese language model is trained by training the recognition results and the labeled scores of the training recognition results.

[0109] In some feasible examples, combining Figure 4 ,like Figure 11 The above-mentioned S17 can be implemented through the following S170 and S171.

[0110] S170. Based on the actual length, predicted length, and theoretical length, determine the first weight value of the candidate recognition result corresponding to the actual length in the fusion score.

[0111] S171. Based on the second weight value, actual score, first weight value and theoretical score of the acoustic features contained in the candidate recognition result in the fusion score, the fusion score of each candidate recognition result is obtained.

[0112] In some feasible examples, combining Figure 4 ,like Figure 12 The above S18 can be specifically implemented through the following S180.

[0113] S180. Based on the fusion score, determine the actual recognition result of the audio data to be recognized as the candidate recognition result corresponding to the maximum value of the fusion score.

[0114] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0115] This application embodiment can divide the server into functional modules according to the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing unit. The integrated modules can be implemented in hardware or as software functional modules. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0116] like Figure 13 As shown in the diagram, an embodiment of this application provides a schematic diagram of the structure of a server 400. It includes a communicator 101 and a processor 102.

[0117] The communicator 101 is configured to acquire audio data to be recognized; the processor 102 is configured to recognize the audio features of the audio data to be recognized acquired by the communicator 101, to obtain at least one acoustic feature corresponding to the audio data to be recognized, and an actual score corresponding to each acoustic feature; the processor 102 is further configured to predict the acoustic features, to determine at least one candidate recognition result containing the acoustic features, and the actual length of the text contained in each candidate recognition result; the processor 102 is further configured to predict the number of characters in the text of the audio data to be recognized acquired by the communicator 101, to obtain the number of characters in the audio data to be recognized. The processor 102 is further configured to perform speech activity detection on the audio data to be recognized acquired by the communicator 101 to obtain the theoretical length of the text contained in the audio data to be recognized; the processor 102 is further configured to score the candidate recognition results to obtain the theoretical score corresponding to each candidate recognition result; the processor 102 is further configured to obtain the fusion score of each candidate recognition result based on the actual score, actual length, predicted length, theoretical length and theoretical score; the processor 102 is further configured to determine the actual recognition result of the audio data to be recognized based on the fusion score.

[0118] In some implementable examples, processor 102 is further configured to input the audio data to be identified acquired by communicator 101 into an audio text word count prediction model to predict the word count of the text and obtain the predicted length of the text contained in the audio data to be identified; wherein, the audio text word count prediction model consists of an object detection algorithm model and a multi-classification model.

[0119] In some implementable examples, processor 102 is further configured to perform speech activity detection on the audio data to be recognized acquired by communicator 101 to determine the tag value corresponding to each audio frame in the audio data to be recognized; processor 102 is further configured to perform speech-to-text operation on the audio data to be recognized acquired by communicator 101 to obtain each character contained in the audio data to be recognized; processor 102 is further configured to align based on tag values ​​and characters to obtain aligned audio frames; processor 102 is further configured to count the audio frames in the aligned audio frames whose tag values ​​are target values ​​to obtain the theoretical length of the text contained in the audio data to be recognized; wherein the text contains at least one character, and one character corresponds to one or more audio frames whose tag values ​​are target values.

[0120] In some implementable examples, processor 102 is further configured to set the label values ​​of audio frames that meet the smoothing conditions in the aligned audio frames to 0, thereby obtaining corrected audio frames; wherein the smoothing conditions include that the label value of the current audio frame is different from the label values ​​of the two adjacent audio frames, and the label values ​​of the two adjacent audio frames are 0; processor 102 is further configured to perform statistics on the audio frames with label values ​​of the target value in the corrected audio frames to obtain the theoretical length of the text contained in the audio data to be identified.

[0121] In some feasible examples, the processor 102 is further configured to input the candidate recognition results into a Chinese language model for scoring, thereby obtaining a theoretical score corresponding to each candidate recognition result; wherein the Chinese language model is trained by training the recognition results and the labeled scores of the training recognition results.

[0122] In some implementable examples, processor 102 is further configured to determine a first weight value in the fusion score of the candidate recognition result corresponding to the actual length based on the actual length, the predicted length, and the theoretical length; processor 102 is further configured to obtain the fusion score of each candidate recognition result based on the second weight value of the acoustic features contained in the candidate recognition result in the fusion score, the actual score, the first weight value, and the theoretical score.

[0123] In some implementable examples, processor 102 is further configured to determine, based on the fusion score, the actual recognition result of the audio data to be recognized as the candidate recognition result corresponding to the maximum value of the fusion score.

[0124] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and their functions will not be repeated here.

[0125] Of course, the server 400 provided in this application embodiment includes, but is not limited to, the modules described above. For example, the server 400 may also include a memory 103. The memory 103 may be used to store the program code of the server 400, and may also be used to store data generated by the server 400 during operation, such as data in write requests.

[0126] As an example, combined Figure 3 The acquisition unit 201 and the sending unit 203 in the server 400 perform the same functions as the communicator 101, the processing unit 202 performs the same functions as the processor 102, and the storage unit 204 performs the same functions as the memory 103.

[0127] like Figure 14As shown, this application embodiment also provides a chip system that can be applied to the server 400 in the foregoing embodiments. The chip system includes at least one processor 1501 and at least one interface circuit 1502. The processor 1501 can be the processor in the server 400. The processor 1501 and the interface circuit 1502 can be interconnected via a line. The processor 1501 can receive and execute computer instructions from the memory of the server 400 through the interface circuit 1502. When the computer instructions are executed by the processor 1501, the server 400 can perform the various steps performed by the server 400 in the foregoing embodiments. Of course, the chip system may also include other discrete devices, which are not specifically limited in this application embodiment.

[0128] This application also provides a computer-readable storage medium for storing computer instructions executed by the server 400 described above.

[0129] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A server, characterized in that, include: The communicator is configured to acquire the audio data to be identified; The processor is configured to identify audio features of the audio data to be identified acquired by the communicator, and obtain at least one acoustic feature corresponding to the audio data to be identified, and an actual score corresponding to each acoustic feature. The processor is further configured to predict the acoustic features, determine at least one candidate recognition result containing the acoustic features, and the actual length of the text contained in each candidate recognition result; The processor is further configured to perform text word count prediction on the audio data to be identified acquired by the communicator, and obtain the predicted length of the text contained in the audio data to be identified. The processor is further configured to perform speech activity detection on the audio data to be identified acquired by the communicator, and obtain the theoretical length of the text contained in the audio data to be identified. The processor is further configured to score the candidate recognition results to obtain a theoretical score for each candidate recognition result; The processor is further configured to determine the first weight value of the candidate recognition result corresponding to the actual length in the fusion score based on the actual length, the predicted length, and the theoretical length; The fusion score for each candidate recognition result is obtained based on the second weight value, actual score, first weight value, and theoretical score of the acoustic features included in the candidate recognition result; wherein the fusion score satisfies the following formula: ； ； in, Indicates acoustic characteristics, Indicates the candidate recognition result i, The fusion score represents the candidate recognition result corresponding to the acoustic features. This represents the second weight value. This represents the sum of the actual scores corresponding to each acoustic feature included in the candidate recognition result i. This represents the third weight value of candidate identification result i in the fusion score. This represents the first weight value corresponding to candidate recognition result i. Let N represent the theoretical score of candidate identification result i, and let N represent the prediction length. Indicates the theoretical length. This represents the actual length of candidate recognition result i. ∈[1, X], where X is the total number of candidate recognition results, and i and X are both integers. and All are constants greater than 0; The processor is also configured to determine the actual recognition result of the candidate recognition result based on the fusion score.

2. The server according to claim 1, characterized in that, The processor is further configured to input the audio data to be identified acquired by the communicator into an audio text word count prediction model to predict the word count of the text and obtain the predicted length of the text contained in the audio data to be identified; wherein the audio text word count prediction model consists of an object detection algorithm model and a multi-classification model.

3. The server according to claim 1, characterized in that, The processor is further configured to perform voice activity detection on the audio data to be identified acquired by the communicator, and determine the tag value corresponding to each audio frame in the audio data to be identified; The processor is further configured to perform a speech-to-text operation on the audio data to be recognized acquired by the communicator to obtain each character contained in the audio data to be recognized; The processor is further configured to align the tag value and the character to obtain an aligned audio frame; The processor is further configured to perform statistics on audio frames with label values ​​of the target value in the aligned audio frames to obtain the theoretical length of the text contained in the audio data to be identified; wherein the text contains at least one character, and one character corresponds to one or more audio frames with label values ​​of the target value.

4. The server according to claim 3, characterized in that, The processor is further configured to set the label value of the audio frames that meet the smoothing condition in the aligned audio frames to 0, thereby obtaining the corrected audio frames; wherein the smoothing condition includes that the label value of the current audio frame is different from the label values ​​of the two adjacent audio frames, and the label values ​​of the two adjacent audio frames are 0. The processor is further configured to perform statistical analysis on audio frames with label values ​​of the target value in the corrected audio frames to obtain the theoretical length of the text contained in the audio data to be identified.

5. The server according to claim 1, characterized in that, The processor is further configured to input the candidate recognition results into a Chinese language model for scoring, thereby obtaining a theoretical score corresponding to each candidate recognition result; wherein the Chinese language model is trained by training the recognition results and the labeled scores of the trained recognition results.

6. The server according to claim 1, characterized in that, The processor is further configured to determine, based on the fusion score, the actual recognition result of the candidate recognition result as the maximum value of the fusion score.

7. A speech recognition method, characterized in that, include: Obtain the audio data to be recognized; The audio features of the audio data to be identified are identified to obtain at least one acoustic feature corresponding to the audio data to be identified, and the actual score corresponding to each acoustic feature; The acoustic features are predicted to determine at least one candidate recognition result containing the acoustic features, and the actual length of the text contained in each candidate recognition result; The text word count of the audio data to be identified is predicted to obtain the predicted length of the text contained in the audio data to be identified. Speech activity detection is performed on the audio data to be identified to obtain the theoretical length of the text contained in the audio data to be identified; The candidate identification results are scored to obtain a theoretical score for each candidate identification result. Based on the actual length, predicted length, and theoretical length, determine the first weight value of the candidate recognition result corresponding to the actual length in the fusion score; The fusion score for each candidate recognition result is obtained based on the second weight value, actual score, first weight value, and theoretical score of the acoustic features included in the candidate recognition result; wherein the fusion score satisfies the following formula: ； ； in, Indicates acoustic characteristics, Indicates the candidate recognition result i, The fusion score represents the candidate recognition result corresponding to the acoustic features. This represents the second weight value. This represents the sum of the actual scores corresponding to each acoustic feature included in the candidate recognition result i. This represents the third weight value of candidate identification result i in the fusion score. This represents the first weight value corresponding to candidate recognition result i. Let N represent the theoretical score of candidate identification result i, and let N represent the prediction length. Indicates the theoretical length. This represents the actual length of candidate recognition result i. ∈[1, X], where X is the total number of candidate recognition results, and i and X are both integers. and All are constants greater than 0; Based on the fusion score, the actual recognition result of the candidate recognition result is determined.

8. The speech recognition method according to claim 7, characterized in that, The step of predicting the number of characters in the audio data to be identified, to obtain the predicted length of the text contained in the audio data to be identified, includes: The audio data to be identified is input into the audio text word count prediction model to predict the word count of the text, thereby obtaining the predicted length of the text contained in the audio data to be identified; wherein, the audio text word count prediction model consists of an object detection algorithm model and a multi-classification model.

9. A computer-readable storage medium, characterized in that, include: The computer-readable storage medium stores a computer program that, when executed by a computing device, causes the computing device to implement the speech recognition method of claim 7 or 8.

Images