A speech recognition method, apparatus and electronic device

Abstract

The present disclosure relates to the technical field of speech recognition, and particularly relates to a speech recognition method, device and electronic device, which are used to solve the problem of how to improve the accuracy of speech recognition of an electronic device. The method comprises: receiving to-be-recognized speech data; recognizing the to-be-recognized speech data to obtain a target recognition result corresponding to the to-be-recognized speech data and a recognition score of the target recognition result; matching the to-be-recognized speech data and background speech data to obtain a matching score between the to-be-recognized speech data and the background speech data; obtaining a fusion score based on the recognition score and the matching score; determining a prompt state of the target recognition result based on the fusion score; and generating prompt information according to the prompt state. The prompt information comprises any one of the target recognition result and the target recognition result whose fusion score is less than a first threshold.

Description

Technical Field

[0001] This disclosure relates to the field of speech recognition technology, and in particular to a speech recognition method, apparatus and electronic device. Background Technology

[0002] In human-computer interaction scenarios, speech recognition plays a crucial role in natural language understanding and generation. The accuracy of speech recognition is both the foundation and the bottleneck for downstream tasks. For example, when users operate electronic devices, they can control them via voice. During this process, the electronic device continuously receives the audio stream corresponding to the user's voice input, decodes the audio stream, and provides real-time feedback on the decoding results. However, there is a problem where the electronic device may misrecognize words when the user is silent. For instance, if the user is not speaking or has a prolonged pause in thought during voice interaction, the electronic device may misrecognize words, resulting in a poor user experience.

[0003] Therefore, improving the accuracy of voice recognition in electronic devices has become an urgent problem to be solved. Summary of the Invention

[0004] To address the aforementioned technical problems, this disclosure provides a speech recognition method, apparatus, and electronic device.

[0005] The technical solution disclosed herein is as follows:

[0006] In a first aspect, this disclosure provides a display device, comprising: a communicator configured to receive voice data to be recognized; wherein one voice data to be recognized corresponds to one user account, and one user account corresponds to background voice data without user voice data; a processor configured to recognize the voice data to be recognized received by the communicator to obtain a target recognition result corresponding to the voice data to be recognized, and a recognition score of the target recognition result; the processor is further configured to match the background voice data and the voice data to be recognized received by the communicator to obtain a matching score between the voice data to be recognized and the background voice data; the processor is further configured to obtain a fusion score based on the recognition score and the matching score; the processor is further configured to determine a prompt state of the target recognition result based on the fusion score; wherein the prompt state includes either a non-display state or a display state; the processor is further configured to generate prompt information according to the prompt state. The prompt information includes either the target recognition result or a target recognition result with a fusion score less than a first threshold.

[0007] Secondly, this disclosure provides a speech recognition method, comprising: receiving speech data to be recognized; wherein, one speech data to be recognized corresponds to one user account, and one user account corresponds to background speech data without user speech data; recognizing the speech data to be recognized to obtain a target recognition result corresponding to the speech data to be recognized, and a recognition score of the target recognition result; matching the background speech data and the speech data to be recognized to obtain a matching score between the speech data to be recognized and the background speech data; obtaining a fusion score based on the recognition score and the matching score; determining a prompting state of the target recognition result based on the fusion score; wherein the prompting state includes either a non-display state or a display state; and generating prompt information according to the prompting state. The prompt information includes either the target recognition result or a target recognition result with a fusion score less than a first threshold.

[0008] Thirdly, this disclosure provides an electronic device, including: a memory and a processor, wherein the memory is used to store a computer program; and the processor is used to cause the electronic device to implement the speech recognition method as provided in any of the second aspects when executing the computer program.

[0009] Fourthly, the present invention provides a computer-readable storage medium, comprising: storing a computer program on the computer-readable storage medium, the computer program being executed by a processor of a speech recognition method as provided in any of the second aspects.

[0010] Fifthly, the present invention provides a computer program product that, when run on a computer, causes the computer to perform a speech recognition method as provided in any of the second aspects.

[0011] It should be noted that the aforementioned computer instructions may be stored, in whole or in part, on the first computer-readable storage medium. The first computer-readable storage medium may be packaged together with the processor of the display device, or it may be packaged separately from the processor of the display device; this disclosure does not impose any limitations on this.

[0012] The descriptions of the second, third, fourth, and fifth aspects in this disclosure can be referenced to the detailed description of the first aspect; and the beneficial effects of the descriptions of the second, third, fourth, and fifth aspects can be referenced to the analysis of the beneficial effects of the first aspect, which will not be repeated here.

[0013] In this disclosure, the names of the aforementioned display devices do not limit the devices or functional modules themselves. In actual implementation, these devices or functional modules may appear under other names. As long as the functions of each device or functional module are similar to those of this disclosure, they fall within the scope of the claims of this disclosure and their equivalents.

[0014] These or other aspects of this disclosure will become more readily apparent in the following description.

[0015] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0016] Using the speech recognition method provided in this disclosure, after the display device receives and recognizes the speech data to be recognized, it first recognizes the speech data to be recognized to obtain the target recognition result corresponding to the speech data to be recognized, and the recognition score of the target recognition result. To prevent speech data without characters from being recognized as a target recognition result containing characters due to recognition errors, the display device first matches the background speech data and the speech data to be recognized when generating prompt information, thereby determining the matching score between the speech data to be recognized and the background speech data without user speech data. Then, based on the matching score and the recognition score, the display device obtains a fusion score. Based on the fusion score, the display device determines the prompt state of the target recognition result, and can then generate prompt information based on the prompt state. For example, when the recognition scores are the same, the higher the matching score between the background speech data and the speech data to be recognized, the higher the matching score, and the higher the corresponding fusion score, meaning a higher probability that the user did not speak. Because the fusion score is higher, the probability that the fusion score is greater than or equal to a first threshold is also higher. Since the prompt state is determined to be "not displayed" when the fusion score is greater than or equal to the first threshold, a prompt message is generated to display the target recognition result where the fusion score is less than the first threshold. Therefore, the display device can identify whether the user spoke when receiving the speech data to be recognized, thus reducing the likelihood of the electronic device recognizing words when the user is silent or has a long pause in thought during voice interaction. Because the problem of silent word recognition is reduced, the accuracy of speech recognition is improved, solving the problem of how to improve the accuracy of speech recognition in electronic devices. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0018] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A schematic diagram illustrating a scenario for the speech recognition method provided in an embodiment of this application;

[0020] Figure 2 This is one of the structural schematic diagrams of the display device provided in the embodiments of this application;

[0021] Figure 3 This is a second schematic diagram of the structure of the display device provided in the embodiments of this application;

[0022] Figure 4 One of the flowcharts of the speech recognition method provided in the embodiments of this application;

[0023] Figure 5 A second schematic flowchart illustrating the speech recognition method provided in this application embodiment;

[0024] Figure 6 The third schematic flowchart of the speech recognition method provided in the embodiments of this application;

[0025] Figure 7 The fourth flowchart illustrating the speech recognition method provided in this application embodiment;

[0026] Figure 8 Fifth schematic flowchart of the speech recognition method provided in the embodiments of this application;

[0027] Figure 9 A flowchart illustrating the speech recognition method provided in this application embodiment is shown in Figure 6.

[0028] Figure 10 Seventh schematic flowchart of the speech recognition method provided in the embodiments of this application;

[0029] Figure 11 This is a schematic diagram of the structure of a display device provided in an embodiment of this application;

[0030] Figure 12 This is a schematic diagram of a chip system provided in an embodiment of this application. Detailed Implementation

[0031] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0032] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0033] The display device provided in this application can have various implementation forms, such as a television, a smart television, a laser projection device, a monitor, an electronic bulletin board, an electronic table, etc. Figure 1 and Figure 2 This is one specific embodiment of the display device of this application.

[0034] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0035] The acoustic model in this embodiment is one of the most important parts of the speech recognition system, and most mainstream systems use Hidden Markov Models for modeling.

[0036] The language model (LM) in this embodiment plays an important role in natural language processing. Its task is to predict the probability of a sentence appearing in a language.

[0037] The Conformer model in this embodiment is a neural network model for natural language processing and speech recognition tasks. Its function is to encode the input text or speech signal into vector form for processing.

[0038] Figure 1 This is a schematic diagram illustrating an operational scenario between a display device and a control device according to one or more embodiments of this application, such as... Figure 1 As shown, a user can operate the display device 200 via a mobile terminal 300 and a control device 100. The control device 100 can be a remote control, and communication between the remote control and the display device includes infrared protocol communication, Bluetooth protocol communication, wireless or other wired methods to control the display device 200. The user can input user commands through buttons on the remote control, voice input, control panel input, etc., to control the display device 200. In some embodiments, a mobile terminal, tablet computer, computer, laptop computer, and other smart devices can also be used to control the display device 200.

[0039] In some embodiments, the electronic device provided in this application can be the aforementioned display device 200. When a user needs to perform voice interaction while using the display device 200, they can press the voice button on the control device 100 and input the voice to be interacted with, or input the voice to be interacted with through another electronic device (such as a mobile phone) that has established a communication connection with the display device 200, or input a wake-up word to the display device 200 to activate its voice interaction function, and then input the voice to be interacted with. Upon receiving the voice to be interacted with, the display device 200 performs preprocessing on the voice (e.g., quantization, sampling, pre-filtering, pre-emphasis, and frame windowing of the audio signal, to remove noise in the search information), thus obtaining the voice data to be recognized. Next, the display device 200 recognizes the speech data to be recognized, obtains the target recognition result corresponding to the speech data to be recognized, and the recognition score of the target recognition result; the display device 200 matches the background speech data and the speech data to be recognized, and obtains the matching score between the speech data to be recognized and the background speech data; the display device 200 obtains the fusion score based on the recognition score and the matching score; the display device 200 determines the prompting state of the target recognition result based on the fusion score; and generates prompting information according to the prompting state.

[0040] In some embodiments, the electronic device provided in this application can be the server 400 described above. When a user needs to perform voice interaction while using the display device 200, they can press the voice button on the control device 100 and input the voice to be interacted with, or input the voice to be interacted with through another electronic device (such as a mobile phone) that has established a communication connection with the display device 200, or input a wake-up word to the display device 200 to activate its voice interaction function, and then input the voice to be interacted with. Upon receiving the voice to be interacted with, the display device 200 performs preprocessing on the voice (e.g., quantization, sampling, pre-filtering, pre-emphasis, and frame windowing of the audio signal, to remove noise in the search information), thus obtaining the voice data to be recognized. Then, the display device 200 sends the voice data to be recognized to the server 400. After receiving the speech data to be recognized from the display device 200, the server 400 recognizes the speech data to obtain the target recognition result and the recognition score of the target recognition result. The server 400 matches the background speech data with the speech data to be recognized to obtain a matching score between the two. Based on the recognition score and the matching score, the server 400 obtains a fusion score. Based on the fusion score, the server 400 determines the prompt status of the target recognition result. Then, based on the prompt status, the server 400 generates prompt information. The server 400 sends display information carrying the prompt information to the display device 200. After receiving the display information sent by the server 400, the display device 200 displays the prompt information according to the prompt information in the display information.

[0041] As described above, the speech recognition method provided in this embodiment first recognizes the speech data to be recognized after obtaining it, obtaining the target recognition result and the recognition score of the target recognition result. To prevent speech data without characters from being recognized as a target recognition result containing characters due to recognition errors, the background speech data and the speech data to be recognized are first matched when displaying the target recognition result, thereby determining the matching score between the speech data to be recognized and the background speech data without user speech data. Then, a fusion score is obtained based on the matching score and the recognition score. Then, based on the matching score and the background speech data, and based on the fusion score, the prompt state of the target recognition result is determined, thereby generating prompt information based on the prompt state. For example, when the recognition scores are the same, the higher the matching score between the background speech data and the speech data to be recognized, the higher the matching score, and the higher the corresponding fusion score, meaning the probability that the user did not speak is higher. Since the fusion score is higher, the probability that the fusion score is greater than or equal to the first threshold is higher. Since the prompt state is set to "not displayed" when the fusion score is greater than or equal to the first threshold, the target recognition result with a fusion score less than the first threshold is displayed instead. Therefore, it can identify whether the user is speaking while receiving the speech data to be recognized, thus reducing the likelihood of the electronic device recognizing words when the user is not speaking or has a long pause in thought during voice interaction with the device. Because the problem of the electronic device recognizing words while silent is reduced, the accuracy of speech recognition is improved.

[0042] Figure 2 A hardware configuration block diagram of a display device 200 according to an exemplary embodiment is shown. For example... Figure 2 The display device 200 shown includes at least one of the following: a tuner / demodulator 210, a communicator 220, a detector 230, an external device interface 240, a controller 250, a display 260, an audio output interface 270, a memory, a power supply, and a user interface 280. The controller includes a central processing unit, a video processor, an audio processor, a graphics processor, RAM, ROM, and a first to nth interface for input / output. The display 260 may be a touch-enabled display, such as a touch screen display. The tuner / demodulator 210 receives broadcast television signals via wired or wireless reception and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals. The detector 230 is used to collect signals from the external environment or signals interacting with the external environment. The controller 250 and the tuner / demodulator 210 may be located in different separate devices; that is, the tuner / demodulator 210 may also be located in an external device of the main device containing the controller 250, such as an external set-top box.

[0043] In some embodiments, the controller 250 controls the operation of the display device and responds to user operations through various software control programs stored in memory. The controller 250 controls the overall operation of the display device 200.

[0044] In some examples, the display device 200 of one or more embodiments is a television set 1, and the operating system of the television set 1 is the Android system, for example... Figure 3 As shown, TV 1 can be logically divided into an application layer (referred to as "application layer") 21, an application framework layer (referred to as "framework layer") 22, an Android runtime and system library layer (referred to as "system runtime library layer") 23, and a kernel layer 24.

[0045] The application layer 21 includes one or more applications. These applications can be system applications or third-party applications. For example, application layer 21 may include a first application that provides speech recognition functionality. The framework layer 22 provides application programming interfaces (APIs) and programming frameworks for the applications in application layer 21. The system runtime library layer 23 provides support for the upper layer, namely the framework layer 22. When the framework layer 22 is used, the Android operating system runs the C / C++ libraries contained in the system runtime library layer 23 to implement the functions required by the framework layer 22. The kernel layer 24 acts as software middleware between the hardware layer and application layer 21, managing and controlling hardware and software resources.

[0046] In some examples, after the TV 1 is turned on, the first application starts. When a user needs to perform a voice search while using the TV 1, they can press the voice button on the control device 100 and input the voice information to be searched, or input the voice information to be searched through other electronic devices (such as mobile phones) that have established a communication connection with the TV 1, or the user can input a wake-up word to the TV 1 to activate the voice interaction function of the TV 1, and then input the voice information to be searched. At this time, the processing unit 101 of the TV 1 determines that after the transceiver unit 102 receives the voice information to be searched, it performs preprocessing on the voice received by the transceiver unit 102 (such as one or more of the following: audio signal quantization, sampling, pre-filtering, pre-emphasis, and frame windowing to remove noise in the search information), thus obtaining the voice data to be recognized. Subsequently, the processing unit 101 of the television set 1 recognizes the speech data to be recognized received by the transceiver unit 102, obtaining the target recognition result corresponding to the speech data to be recognized, and the recognition score of the target recognition result; the processing unit 101 of the television set 1 matches the background speech data with the speech data to be recognized received by the transceiver unit 102, obtaining the matching score between the speech data to be recognized and the background speech data; the processing unit 101 of the television set 1 obtains a fusion score based on the recognition score and the matching score; the processing unit 101 of the television set 1 determines the prompt status of the target recognition result based on the fusion score. The processing unit 101 of the television set 1 generates prompt information according to the prompt status in the displayed information.

[0047] Specifically, when the executing entity of the speech recognition method provided in this embodiment is a television set 1, when the television set 1 receives the voice input by the user that requires voice interaction, the television set 1 will divide the voice into segments of audio and transmit each segment of audio as a buffer packet, thereby forming the speech data to be recognized.

[0048] Specifically, when the executing entity of the speech recognition method provided in this embodiment is a television set 1, the storage unit 103 of the television set 1 is used to store data such as the program code of the speech recognition method provided in this embodiment.

[0049] In some examples, after the TV 1 is turned on, the first application starts. When a user needs to perform a voice search while using the TV 1, they can press the voice button on the control device 100 and input the voice message to search for information, or input the voice message through another electronic device (such as a mobile phone) that has established a communication connection with the TV 1, or input a wake-up word to the TV 1 to activate its voice interaction function, and then input the voice message to interact with the TV 1. At this time, the processing unit 101 of the TV 1 determines that after the transceiver unit 102 receives the voice message to interact with the TV 1, it performs preprocessing on the received voice message (e.g., quantization, sampling, pre-filtering, pre-emphasis, and frame windowing of the audio signal, to remove noise in the search information), thus obtaining the voice data to be recognized. Afterwards, the processing unit 101 of the TV 1 controls the transceiver unit 102 to send the voice data to be recognized to the server 400. The transceiver unit 201 of the server 400 receives the voice data to be recognized sent by the TV 1. The processing unit 202 of server 400 recognizes the speech data to be recognized received by transceiver unit 201, obtaining the target recognition result corresponding to the speech data to be recognized, and the recognition score of the target recognition result; the processing unit 202 of server 400 matches the background speech data with the speech data to be recognized received by transceiver unit 201, obtaining the matching score between the speech data to be recognized and the background speech data; the processing unit 202 of server 400 obtains a fusion score based on the recognition score and the matching score; the processing unit 202 of server 400 determines the prompt status of the target recognition result based on the fusion score. The processing unit 202 of server 400 generates prompt information based on the prompt status. Then, the processing unit 202 of server 400 controls transceiver unit 201 to send display information carrying the prompt information to display device 200. After the processing unit 101 of television set 1 determines that transceiver unit 102 has received the display information sent by server 400, it displays the prompt information according to the prompt information in the display information.

[0050] Specifically, when the execution entity of the speech recognition method provided in this embodiment is the server 400, when the television 1 receives the user's input of speech requiring voice interaction, the television 1 divides the speech into audio segments and uploads each audio segment to the server 400 via the network in the form of a buffer packet. Subsequently, the server 400 continuously receives buffer packets sent by the display device 200, thereby forming the speech data to be recognized.

[0051] Specifically, when the execution subject of the speech recognition method provided in this embodiment is a server 400, the storage unit 203 of the server 400 is used to store data such as the program code of the speech recognition method provided in this embodiment.

[0052] It should be noted that when the executing entity is TV set 1, either TV set 1 or server 400 can perform preprocessing of the voice; there is no limitation here. Similarly, when the executing entity is server 400, either TV set 1 or server 400 can perform preprocessing of the voice; there is no limitation here.

[0053] In the following embodiments, the television set 1 described above is used as the execution subject for the speech recognition method provided in the embodiments of this disclosure to illustrate the method of this application.

[0054] This application provides a speech recognition method applied to a television set 1, such as... Figure 4 As shown, the speech recognition method may include S11-S16.

[0055] S11. Receive the voice data to be recognized. The voice data to be recognized is received by the display device in response to the activation operation of the semantic interaction function. One voice data to be recognized corresponds to one user account, and one user account corresponds to one background voice data without user voice data.

[0056] In some examples, during voice interaction between a user and television set 1, television set 1 may fail to accurately recognize the user's input voice data multiple times (e.g., twice or more). To improve the accuracy of semantic recognition, the voice recognition method provided in this embodiment prompts the user to perform a target operation when the user's input voice data cannot be accurately recognized multiple times. The target operation may include closing doors or windows. Subsequently, in response to the user's action of performing the target operation, television set 1 prompts the user not to make any sound and acquires background voice data without user voice data through an audio acquisition device (e.g., a microphone) installed on television set 1. Thus, background voice data without user voice data is obtained.

[0057] Furthermore, during use, if TV 1 has already recorded background voice data but still fails to accurately recognize the user's input voice data multiple times, TV 1 can prompt the user to perform the target operation again. Then, in response to the user's action of performing the target operation, TV 1 prompts the user to remain silent and acquires background voice data without user voice data through an audio acquisition device (such as a microphone) installed on TV 1. In this way, updated background voice data is obtained.

[0058] In some examples, to reduce the computational resource consumption of the television 1, the speech recognition method provided in this disclosure executes the speech recognition method in response to a user's activation of the semantic interaction function, such as pressing the voice button on the remote control. When the user releases the voice button, the television 1 releases the computational resources consumed by the speech recognition method. This process continues until the user activates the semantic interaction function again, at which point the television 1 again executes the speech recognition method in response to the press of the voice button.

[0059] In some examples, the user account can be a user-pre-registered account that is logged in on TV 1, or an anonymous account that is uniquely assigned to TV 1 by server 400 when the user logs in on TV 1. There is no limitation here.

[0060] S12. Recognize the speech data to be recognized to obtain the target recognition result corresponding to the speech data to be recognized, and the recognition score of the target recognition result. The target recognition result may contain at least one character, or it may not contain any characters.

[0061] In some examples, when recognizing the speech data to be recognized, the audio features of the audio data to be recognized can be identified to obtain at least one acoustic feature corresponding to the audio data to be recognized. Then, the acoustic features are predicted to determine at least one candidate recognition result containing the acoustic features, and a recognition score for each candidate recognition result.

[0062] In some examples, the speech data to be recognized can be input into a feature extractor to extract audio features (such as Mel Frequency Cepstrum Coefficients (MFCCs)) to obtain at least one audio feature. The audio feature is then stored in a feature queue, awaiting decoding.

[0063] In some examples, the training process for the feature extractor is as follows:

[0064] Obtain training sample data and the labeling results of the training sample data. The training sample data includes historical recognition audio streams, which include historical recognition audio streams with user voices and historical recognition audio streams without user voices. The labeling results include the audio features corresponding to the historical recognition audio streams.

[0065] The training sample data is input into the neural network model for learning, and the prediction results of the neural network model on the training sample data are obtained.

[0066] Based on the prediction and labeling results, the network parameters of the neural network model are adjusted until the neural network model converges, thus obtaining the feature extractor.

[0067] In some examples, when identifying the audio features of the audio stream to be identified, audio features can be extracted first to obtain the corresponding audio features (such as Mel Frequency Cepstrum Coefficients (MFCCs) and filter bank (Fbank)). Then, the audio features are identified to obtain the acoustic features corresponding to the audio stream to be identified.

[0068] In some examples, audio features can be input into a pre-trained speech model for recognition to obtain the acoustic features corresponding to the audio stream to be recognized, as well as the actual scores corresponding to the acoustic features.

[0069] In some examples, the training process for the pre-trained acoustic model is as follows:

[0070] Obtain training sample data and the labeling results of the training sample data. The training sample data includes historical audio streams, which include historical audio streams with user voices and historical audio streams without user voices, as well as the audio features corresponding to each historical audio stream. The labeling results include the acoustic features corresponding to the audio features of the historical audio streams.

[0071] The training sample data is input into a neural network model (such as the Conformer model) for learning, and the neural network model's prediction results on the training sample data are obtained.

[0072] Based on the prediction and labeling results, the network parameters of the neural network model are adjusted until the neural network model converges, resulting in a pre-trained acoustic model.

[0073] In some examples, when predicting acoustic features, the acoustic features can be fed into a pre-trained language model to obtain at least one result to be identified that includes the acoustic features.

[0074] In some examples, the training process of a pre-trained language model is as follows:

[0075] Obtain training sample data and the labeling results of the training sample data. The training sample data includes historical audio streams, which include historical audio streams with user voices and historical audio streams without user voices, as well as the acoustic features corresponding to each historical audio stream. The labeling results include at least one identification result corresponding to the acoustic features of the historical audio streams, and the identification score of each identification result.

[0076] The training sample data is input into a neural network model (e.g., LM) for learning, and the neural network model's prediction results on the training sample data are obtained.

[0077] Based on the prediction and labeling results, the network parameters of the neural network model are adjusted until the neural network model converges, resulting in a pre-trained language model.

[0078] In some examples, a pre-trained language model can predict the probability of the target text corresponding to a given acoustic feature appearing in the language (such as a word or sentence) containing that target text, thus obtaining the probability of the target result for each language. One language corresponds to one target result. Then, the target result with the highest probability of appearance is taken as the target recognition result.

[0079] In some examples, by introducing data without user voice, the pre-trained acoustic model and pre-trained language model will recognize the data without sound during the training process, thus generating target recognition results that do not contain characters, reducing the probability of silent characters appearing.

[0080] S13. Match the background speech data and the speech data to be recognized to obtain the matching score between the speech data to be recognized and the background speech data.

[0081] In some examples, a similarity score can be calculated between the background speech data and the speech data to be recognized. This similarity score is then used as the matching score between the speech data to be recognized and the background speech data.

[0082] In some examples, a first audio feature can be extracted from the background speech data, and a first feature vector can be obtained based on the first audio feature. Then, a second audio feature is extracted from the speech data to be recognized, and a second feature vector is obtained based on the second audio feature. Finally, the similarity between the background speech data and the speech data to be recognized is obtained by calculating the similarity between the first and second feature vectors (such as cosine similarity, Euclidean distance, etc.).

[0083] S14. Based on the recognition score and the matching score, obtain the fusion score.

[0084] In some examples, the fusion score can be obtained based on the sum of the product of the recognition score and the first weight, and the product of the matching score and the second weight. Here, the sum of the first and second weights equals 1.

[0085] S15. Based on the fusion score, determine the prompting state of the target recognition result. The prompting state includes either a no-display state or a display state.

[0086] S16. Generate prompt information according to the prompt status. The prompt information includes either the target recognition result or the target recognition result with a fusion score less than the first threshold.

[0087] In some examples, different fusion scores correspond to different prompt states. For instance, when the fusion score is less than a first threshold, the prompt state is determined to be a display state. In this case, when the prompt state is display state, TV 1 generates a prompt message containing the target recognition result. Then, TV 1 displays this prompt message. Alternatively, when the fusion score is greater than or equal to the first threshold, the prompt state is determined to be a non-display state. In this case, when the prompt state is non-display state, TV 1 generates a prompt message containing the target recognition result with a fusion score less than the first threshold. Then, TV 1 displays this prompt message.

[0088] In some embodiments, the no-display state includes delayed waiting and no text; when the fusion score is greater than or equal to a first threshold and less than a second threshold, the prompt state is determined to be delayed waiting; wherein the first threshold is less than the second threshold; when the prompt state is delayed waiting, a prompt message containing the target recognition result that the fusion score is less than the first threshold is generated. Then, the television set 1 displays this prompt message.

[0089] Alternatively, if the fusion score is greater than or equal to the second threshold, the prompt status is determined to be "no text"; if the prompt status is "no text", a prompt message containing the target recognition result with a fusion score less than the first threshold is generated. Then, television set 1 displays this prompt message.

[0090] It should be noted that the prompt information for the target recognition result with a fusion score less than the first threshold includes: the target recognition result with a fusion score less than the first threshold in the voice data to be recognized received by TV 1 after the last response to the start operation of the semantic interaction function.

[0091] In some examples, during a user's voice conversation with TV 1, the input voice data to be recognized is continuous. To reduce the computational resource consumption of TV 1, when the number of consecutive times the prompt status is displayed is greater than or equal to a threshold, it indicates that the user is continuously inputting voice data to be recognized. Therefore, the calculation of the matching score can be paused, that is, after obtaining the target recognition result corresponding to the voice data to be recognized, the target recognition result is directly displayed.

[0092] In some examples, to prevent the issue of silent output during the pause of matching score calculation, the voice recognition method provided in this embodiment releases the computing resources occupied by the voice recognition method in response to the user releasing the voice button on the remote control. Until the user re-activates the semantic interaction function, the television 1 again responds to the press of the voice button on the remote control and executes the voice recognition method provided in this embodiment, i.e., recalculating the matching score and recognition score.

[0093] As described above, in order to prevent speech data without characters from being recognized as containing characters due to recognition errors, the display device first matches the background speech data and the speech data to be recognized when generating prompt information, thereby determining the matching score between the speech data to be recognized and the background speech data without user speech data. Then, the display device obtains a fusion score based on the matching score and the recognition score. Based on the fusion score, the display device determines the prompt state of the target recognition result, and can generate prompt information based on the prompt state. For example, when the recognition scores are the same, the higher the matching score between the background speech data and the speech data to be recognized, the higher the matching score, and the higher the corresponding fusion score, meaning a higher probability that the user did not speak. Since the fusion score is higher, the probability that the fusion score is greater than or equal to a first threshold is higher. When the fusion score is greater than or equal to the first threshold, the prompt state is determined to be a non-display state, and a prompt message is generated indicating that the target recognition result with a fusion score less than the first threshold is displayed. Therefore, the display device can identify whether the user has spoken when receiving the voice data to be recognized. This reduces the likelihood of the electronic device recognizing words when the user is silent or has paused for a long time during voice interaction. Since the problem of silent speech is reduced, the accuracy of voice recognition is improved.

[0094] In some feasible examples, combining Figure 4 ,like Figure 5 As shown, the above S12 can be implemented by the following S120 and S121.

[0095] S120. Recognize the speech data to be recognized to obtain at least one recognition result corresponding to the speech data to be recognized, and a recognition score for each recognition result.

[0096] S121. Based on the recognition score, determine the target recognition result as the result corresponding to the highest recognition score.

[0097] In some feasible examples, combining Figure 4 ,like Figure 6 As shown, the above S13 can be implemented by the following S130 and S131.

[0098] S130. Calculate the similarity between the speech data to be recognized and the background speech data to obtain the similarity between the speech data to be recognized and the background speech data.

[0099] S131. Based on similarity, determine the matching score between the speech data to be identified and the background speech data.

[0100] In some feasible examples, combining Figure 4 ,like Figure 7 As shown, the above S14 can be implemented in the following S140.

[0101] S140. The fusion score is obtained by summing the product of the recognition score and the first weight, and the product of the matching score and the second weight.

[0102] In some feasible examples, combining Figure 4 ,like Figure 8 As shown, the above S15 can be implemented by the following S150.

[0103] S150. When the fusion score is less than the first threshold, the prompt state is determined to be the display state.

[0104] In some feasible examples, combining Figure 4 ,like Figure 8 As shown, the above S15 can be implemented in the following S151.

[0105] S151. When the fusion score is greater than or equal to the first threshold, the prompt status is determined to be "not displayed".

[0106] In some feasible examples, not displaying the state includes delayed waiting; combined with Figure 8 ,like Figure 9 As shown, S151 can be implemented by S1510, and S16 can be implemented by S160.

[0107] S1510. When the fusion score is greater than or equal to the first threshold and less than the second threshold, determine the prompt status as delayed waiting. Wherein, the first threshold is less than the second threshold.

[0108] S160. When the prompt status is delayed waiting, generate a prompt message containing the target recognition result with a fusion score less than the first threshold.

[0109] In some feasible examples, not displaying a state includes having no text; combined with Figure 8 ,like Figure 9 As shown, S151 can be implemented by S1511 as described below, and S16 can be implemented by S161 as described below.

[0110] S1511. When the fusion score is greater than or equal to the second threshold, determine the prompt status as "no text".

[0111] S161. When the prompt status is "no text", generate a prompt message indicating that the target recognition result with a fusion score is less than the first threshold.

[0112] In some feasible examples, combining Figure 4 ,like Figure 10 As shown, the speech recognition method provided in this embodiment of the present disclosure further includes: S17.

[0113] S17. If the number of consecutive times the prompt status is displayed is greater than or equal to the number of times the prompt threshold is reached, stop calculating the matching score.

[0114] In some feasible examples, the voice data to be identified is received by the display device in response to an activation operation of the semantic interaction function.

[0115] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0116] This application embodiment can divide the display device into functional modules according to the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing unit. The integrated modules can be implemented in hardware or as software functional modules. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0117] like Figure 11 As shown in the diagram, an embodiment of this application provides a schematic diagram of the structure of a display device 200. It includes a communicator 101 and a processor 102.

[0118] The communicator 101 is configured to receive voice data to be recognized; wherein one voice data to be recognized corresponds to one user account, and one user account corresponds to background voice data without user voice data; the processor 102 is configured to recognize the voice data to be recognized received by the communicator 101, and obtain a target recognition result corresponding to the voice data to be recognized, and a recognition score for the target recognition result; wherein the target recognition result contains at least one character, or the target recognition result does not contain any character; the processor 102 is further configured to match the background voice data and the voice data to be recognized received by the communicator 101, and obtain a matching score between the voice data to be recognized and the background voice data; the processor 102 is further configured to obtain a fusion score based on the recognition score and the matching score; the processor 102 is further configured to determine a prompt state for the target recognition result based on the fusion score; wherein the prompt state includes either a non-display state or a display state; the processor 102 is further configured to generate prompt information according to the prompt state. The prompt information includes either the target recognition result or a target recognition result with a fusion score less than a first threshold.

[0119] In some implementable examples, processor 102 is further configured to recognize the speech data to be recognized received by communicator 101 to obtain at least one recognition result corresponding to the speech data to be recognized, and a recognition score for each recognition result; processor 102 is further configured to determine, based on the recognition score, the recognition result corresponding to the highest recognition score as the target recognition result.

[0120] In some implementable examples, processor 102 is further configured to perform a similarity calculation on background speech data and speech data to be identified received by communicator 101 to obtain a similarity between speech data to be identified and background speech data; processor 102 is further configured to determine a matching score between speech data to be identified and background speech data based on the similarity.

[0121] In some implementable examples, processor 102 is further configured to obtain a fusion score based on the sum of the product of the recognition score and the first weight, and the product of the matching score and the second weight.

[0122] In some implementable examples, processor 102 is further configured to determine the prompt state as a display state when the fusion score is less than a first threshold.

[0123] In some implementable examples, processor 102 is further configured to determine the prompt state as a non-display state when the fusion score is greater than or equal to a first threshold.

[0124] In some implementable examples, the state of not displaying includes a delayed waiting state; the processor 102 is further configured to determine the prompt state as delayed waiting state when the fusion score is greater than or equal to a first threshold and the fusion score is less than a second threshold; wherein the first threshold is less than the second threshold; the processor 102 is further configured to generate a prompt message containing a target recognition result with a fusion score less than the first threshold when the prompt state is delayed waiting state.

[0125] In some implementable examples, the no-display state includes no text; processor 102 is further configured to determine the prompt state as no text when the fusion score is greater than or equal to a second threshold; processor 102 is further configured to generate prompt information containing a target recognition result with a fusion score less than a first threshold when the prompt state is no text.

[0126] In some implementable examples, processor 102 is also configured to stop calculating the matching score if the number of consecutive times the prompt status is displayed is greater than or equal to a threshold.

[0127] In some feasible examples, the voice data to be identified is received by the display device in response to an activation operation of the semantic interaction function.

[0128] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and their functions will not be repeated here.

[0129] Of course, the display device 200 provided in this application embodiment includes, but is not limited to, the modules described above. For example, the display device 200 may also include a memory 103. The memory 103 may be used to store the program code of the display device 200, and may also be used to store data generated by the display device 200 during operation, such as data in write requests.

[0130] As an example, combined Figure 3The transceiver unit 102 in the display device 200 performs the same function as the communicator 101, the processing unit 101 performs the same function as the processor 102, and the storage unit 103 performs the same function as the memory 103.

[0131] like Figure 12 As shown, this application embodiment also provides a chip system that can be applied to the display device 200 in the foregoing embodiments. The chip system includes at least one processor 1501 and at least one interface circuit 1502. The processor 1501 may be the processor in the display device 200. The processor 1501 and the interface circuit 1502 are interconnected via a circuit. The processor 1501 can receive and execute computer instructions from the memory of the display device 200 through the interface circuit 1502. When the computer instructions are executed by the processor 1501, the display device 200 can perform the various steps performed by the display device 200 in the foregoing embodiments. Of course, the chip system may also include other discrete devices, which are not specifically limited in this application embodiment.

[0132] This application also provides a computer-readable storage medium for storing computer instructions executed by the display device 200.

[0133] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A display device, characterized in that, include: A communicator is configured to receive voice data to be recognized; wherein one voice data to be recognized corresponds to one user account, and one user account corresponds to background voice data without user voice data. The processor is configured to recognize the speech data to be recognized received by the communicator, and obtain a target recognition result corresponding to the speech data to be recognized, and a recognition score of the target recognition result; The processor is further configured to match the background speech data and the speech data to be recognized received by the communicator to obtain a matching score between the speech data to be recognized and the background speech data. The processor is further configured to obtain a fusion score based on the recognition score and the matching score; The processor is further configured to determine the prompting state of the target recognition result based on the fusion score; wherein the prompting state includes either a non-display state or a display state; The processor is further configured to generate prompt information according to the prompt state; wherein the prompt information includes either the target recognition result or the target recognition result with a fusion score less than a first threshold.

2. The display device according to claim 1, characterized in that, The processor is further configured to recognize the voice data to be recognized received by the communicator, and obtain at least one recognition result corresponding to the voice data to be recognized, and a recognition score for each recognition result. The processor is further configured to determine, based on the recognition score, the recognition result corresponding to the highest recognition score as the target recognition result.

3. The display device according to claim 1, characterized in that, The processor is further configured to perform a similarity calculation on the background speech data and the speech data to be identified received by the communicator, to obtain the similarity between the speech data to be identified and the background speech data; The processor is further configured to determine a matching score between the speech data to be identified and the background speech data based on the similarity.

4. The display device according to claim 1, characterized in that, The processor is further configured to obtain a fusion score based on the sum of the product of the recognition score and the first weight, and the product of the matching score and the second weight.

5. The display device according to claim 1, characterized in that, The processor is further configured to determine that the prompt state is a display state when the fusion score is less than the first threshold.

6. The display device according to claim 1, characterized in that, The processor is further configured to determine that the prompt state is a non-display state when the fusion score is greater than or equal to a first threshold.

7. The display device according to claim 6, characterized in that, The non-display state includes a delayed waiting state; the processor is further configured to determine the prompt state as a delayed waiting state when the fusion score is greater than or equal to the first threshold and the fusion score is less than the second threshold; wherein the first threshold is less than the second threshold; The processor is further configured to generate a prompt message containing a target recognition result where the fusion score is less than the first threshold when the prompt state is a delayed waiting state.

8. The display device according to claim 7, characterized in that, The non-display state includes no text; the processor is further configured to determine that the prompt state is no text when the fusion score is greater than or equal to the second threshold. The processor is further configured to generate a prompt message containing a target recognition result where the fusion score is less than the first threshold when the prompt status is blank.

9. The display device according to claim 1, characterized in that, The processor is also configured to stop calculating the matching score if the number of consecutive times the prompt state is displayed is greater than or equal to a number threshold.

10. The display device according to any one of claims 1-9, characterized in that, The voice data to be recognized is received by the display device in response to the activation of the semantic interaction function.

Images