Voice processing method, device, storage medium and electronic device

Abstract

The embodiment of the present invention provides a method, device, storage medium and electronic device for speech processing, the method comprising: when the speech processing result corresponding to the speech segment collected by the terminal device meets the result abnormality condition, obtaining the speech to be recognized from the target storage area of ​​the server, wherein the speech to be recognized includes the speech segment, and the speech processing result includes the result obtained by speech recognition or semantic analysis of the speech segment; performing speech recognition verification based on the speech to be recognized to obtain a first verification result; according to the result type corresponding to the first verification result, controlling the terminal device to perform a setting operation, and / or performing a second verification process on the speech to be recognized based on the first verification result. The problem that speech recognition processing is prone to errors existing in the related art is solved, resulting in low accuracy of speech recognition and low user experience.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of smart home, and more specifically, to a method, device, storage medium and electronic device for voice processing. Background Art

[0002] With the advancement of science and technology and the vigorous development of artificial intelligence voice, more and more terminal devices with voice functions are applied to daily life, such as smart voice speakers, smart voice air conditioners, smart voice TVs, smart voice curtains, etc. These devices with voice interaction functions have facilitated people's lives to a certain extent.

[0003] In the related art, there are still many problems in the process of voice interaction with devices with voice interaction functions, such as false wake-up, single-round voice interaction recognition errors, multi-round voice interaction recognition errors, etc. Therefore, the occurrence of these voice interaction errors will greatly reduce the user experience.

[0004] Currently, no effective solution has been proposed to address the problem that speech recognition processing in related technologies is prone to errors, resulting in low speech recognition accuracy and low user experience. Summary of the invention

[0005] The embodiments of the present invention provide a method, device, storage medium and electronic device for speech processing, so as to at least solve the problem that speech recognition processing is prone to errors in the related art, thereby resulting in low accuracy of speech recognition and low user experience.

[0006] According to one embodiment of the present invention, a method for speech processing is provided, comprising: when a speech processing result corresponding to a speech segment collected by a terminal device satisfies a result abnormality condition, obtaining a speech to be recognized from a target storage area of ​​a server, wherein the speech to be recognized includes the speech segment, and the speech processing result includes a result obtained by performing speech recognition or semantic analysis on the speech segment; performing speech recognition verification based on the speech to be recognized to obtain a first verification result; according to a result type corresponding to the first verification result, controlling the terminal device to execute a setting operation, and / or performing a second verification process on the speech to be recognized based on the first verification result.

[0007] In an exemplary embodiment, the speech processing result includes a speech recognition result and a semantic analysis result, and the method further includes: when the speech recognition result corresponding to the speech segment satisfies any one of the following conditions: the speech recognition result is empty, a boundary detection error occurs, and the speech recognition result fails to match, determining that the speech processing result corresponding to the speech segment meets the result abnormality condition, wherein the speech recognition result matching failure indicates that no recognition statement matching the speech recognition result is found in the first database; and / or when the semantic analysis result corresponds to the semantic analysis result matching failure, determining that the speech processing result corresponding to the speech segment meets the result abnormality condition, wherein the semantic analysis result is the result obtained by the semantic processing module performing semantic analysis processing on the speech recognition result, and the semantic analysis result matching failure indicates that no operation intention matching the semantic analysis result is found in the second database.

[0008] In an exemplary embodiment, the speech to be recognized is the original speech collected by the terminal device, the speech segment is part of the audio in the original speech, and the terminal device is controlled to perform a setting operation according to the result type corresponding to the first verification result, including: when it is determined that the result type represents that the speech to be recognized is a wake-up type speech, the terminal device is controlled to perform a wake-up operation and prohibit the first verification result from being sent to the semantic processing module; or, when it is determined that the result type represents that the speech to be recognized is a noise type audio, the terminal device is controlled to perform a sound pickup operation and prohibit the first verification result from being sent to the semantic processing module.

[0009] In an exemplary embodiment, the second verification processing of the speech to be recognized based on the first verification result includes: when the result type of the first verification result indicates that the speech to be recognized is empty and / or a boundary detection error occurs, sending the first verification result and the speech to be recognized to an error diagnosis system; and using the error diagnosis system, performing the second verification processing on the original speech based on the first verification result and the speech to be recognized.

[0010] In an exemplary embodiment, the second verification process of the speech to be recognized based on the first verification result includes: when the result type of the first verification result indicates that the semantic recognition result of the speech to be recognized is normal, performing semantic analysis on the first verification result to obtain the semantic analysis result of the speech to be recognized; matching the semantic analysis result of the speech to be recognized with the operation intention in a second database; in case of a matching failure, sending the semantic analysis result of the speech to be recognized, the matching failure result and the speech to be recognized to an error diagnosis system; and using the error diagnosis system, performing the second verification process on the speech to be recognized based on the semantic analysis result of the speech to be recognized, the matching failure result and the speech to be recognized.

[0011] In an exemplary embodiment, the method also includes: in case of successful matching, obtaining an operation instruction corresponding to the operation intention; controlling the terminal device to update the stored second speech recognition model based on the first speech recognition model stored in the server, wherein the server uses the first speech recognition model for speech recognition, and the terminal device uses the second speech recognition model for speech recognition.

[0012] In an exemplary embodiment, the method further includes: deleting the speech to be recognized stored in the target storage area when no query operation on the speech to be recognized in the target storage area is received within a predetermined time period.

[0013] In an exemplary embodiment, the method further includes: deleting the speech to be recognized stored in the target storage area when no query operation for the speech to be recognized in the target storage area is received within a predetermined time period. An acquisition module is used to acquire the speech to be recognized from the target storage area of ​​the server when the speech processing result corresponding to the speech segment collected by the terminal device meets the result abnormality condition, wherein the speech to be recognized includes the speech segment, and the speech processing result includes the result obtained by performing speech recognition or semantic analysis on the speech segment; a recognition module is used to perform speech recognition verification based on the speech to be recognized to obtain a first verification result; a processing module is used to control the terminal device to perform a setting operation according to the result type corresponding to the first verification result, and / or perform a second verification process on the speech to be recognized based on the first verification result.

[0014] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when run.

[0015] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0016] Through the present invention, when the speech processing result corresponding to the speech segment collected by the terminal device meets the result abnormality condition, the speech to be recognized can be obtained from the target storage area of ​​the server, wherein the speech to be recognized includes the speech segment, and the speech processing result includes the result obtained by performing speech recognition or semantic analysis on the speech segment, so that the speech to be recognized can be verified by speech recognition to obtain a first verification result, and then, according to the result type corresponding to the first verification result, the terminal device is controlled to perform a setting operation, and / or the speech to be recognized is subjected to a second verification process based on the first verification result. By adopting the method of the present invention, the speech to be recognized can be stored in the target storage area, thereby reducing the probability of speech interaction errors by performing a secondary recognition verification on the stored speech to be recognized. This solves the problem that speech recognition processing is prone to errors in the related art, resulting in low accuracy of speech recognition and low user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 is a hardware structure block diagram of a mobile terminal of the voice processing method according to an embodiment of the present invention;

[0020] Figure 2 is a flowchart of a method for speech processing according to an embodiment of the present invention;

[0021] Figure 3 is an overall flow chart of a speech recognition verification system according to an embodiment of the present invention;

[0022] Figure 4 is a structural block diagram of a speech processing apparatus according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] According to one aspect of an embodiment of the present application, a method for speech processing is provided. The method for speech processing is widely used in smart home, smart home, smart home device ecology, smart house ecology and other whole-house intelligent digital control application scenarios. Optionally, in this embodiment, the above-mentioned method for speech processing can be applied to Figure 1 In the hardware environment composed of the terminal device 102 and the server 104 shown in FIG. Figure 1 As shown, the server 104 is connected to the terminal device 102 via a network, and can be used to provide services (such as application services, etc.) for the terminal or a client installed on the terminal. A database can be set on the server or independently of the server to provide data storage services for the server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data computing services for the server 104.

[0026] The network may include but is not limited to at least one of the following: wired network, wireless network. The wired network may include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the wireless network may include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal device 102 may be but is not limited to a PC, a mobile phone, a tablet computer, a smart air conditioner, a smart range hood, a smart refrigerator, a smart oven, a smart stove, a smart washing machine, a smart water heater, a smart washing device, a smart dishwasher, a smart projection device, a smart TV, a smart clothes drying rack, a smart curtain, a smart audio and video, a smart socket, a smart speaker, a smart fresh air device, a smart kitchen and bathroom device, a smart bathroom device, a smart sweeping robot, a smart window cleaning robot, a smart mopping robot, a smart air purification device, a smart steamer, a smart microwave oven, a smart kitchen treasure, a smart purifier, a smart water dispenser, a smart door lock, etc.

[0027] In this embodiment, a method for speech processing is provided. Figure 2 is a flow chart of a method for speech processing according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0028] Step S202, when the speech processing result corresponding to the speech segment collected by the terminal device meets the result abnormality condition, obtaining the speech to be recognized from the target storage area of ​​the server, wherein the speech to be recognized includes the speech segment, and the speech processing result includes the result obtained by performing speech recognition or semantic analysis on the speech segment;

[0029] Step S204, performing speech recognition verification based on the speech to be recognized to obtain a first verification result;

[0030] Step S206: Control the terminal device to execute a setting operation according to the result type corresponding to the first verification result, and / or perform a second verification process on the speech to be recognized based on the first verification result.

[0031] Among them, the executor of the above steps can be an intelligent processing device with voice error correction processing capabilities, or an intelligent system with voice error correction processing capabilities, a controller, or a controller or processor included in the intelligent system, or it can also be a control device or processing device independently set up in the intelligent system, or it can also be other processing devices or processing units with similar processing capabilities.

[0032] In the above embodiment, the speech processing results can be divided into normal results and abnormal results, and the abnormal results can be of various types, for example, they can be empty, wake-up words, environmental noise, VAD (Voice Activity Detection, speech boundary detection) errors, etc., wherein the wake-up words can be Xiaobai, Xiaoai, Xiaomi, etc., the environmental noise can be the crying of infants and young children, the sound of TV playing movies or TV series, the surrounding construction noise, etc., the VAD error can be incomplete sentence recognition, sentence segmentation recognition errors, etc., for example, when the speech to be recognized is "Xiaobai, I want to watch a movie", when "I want to watch a movie" is recognized, "electricity" may be recognized as the beginning of the speech, and "film" may be recognized as the end of the speech, resulting in incomplete sentence recognition, when the speech to be recognized is "this apple is not very delicious", when "this apple is not very delicious" is recognized, it may be recognized as "this apple is not very delicious", or it may be recognized as "this apple is not big, delicious", resulting in sentence segmentation recognition errors, it should be noted that the above-mentioned examples of abnormal result types are only exemplary embodiments, and the above-mentioned abnormal result types are not limited to the above-mentioned examples.

[0033] In the above embodiment, there may be multiple voices to be recognized in the target storage area, and when multiple voice processing results corresponding to multiple voice segments collected by the terminal device meet the result abnormality condition, multiple voices to be recognized can be obtained from the target storage area of ​​the server, and then the multiple voice segments included in the multiple voices to be recognized can be recognized and processed in turn, and the multiple voice segments included in the multiple voices to be recognized can also be recognized and processed at the same time. In addition, according to the time sequence in which the multiple voices to be recognized are stored in the target storage area, the voices to be recognized with earlier recognition processing time can be given priority, and according to the importance of each of the multiple voices to be recognized, the more important voices to be recognized can be given priority. In addition, there can be multiple terminal devices, so multiple terminal devices can be used to collect voice segments included in the voices to be recognized, and so on.

[0034] In the above embodiment, when the speech processing result corresponding to the speech segment collected by the terminal device meets the result abnormality condition, the speech to be recognized can be obtained from the target storage area of ​​the server, wherein the speech to be recognized includes the speech segment, and the speech processing result includes the result obtained by performing speech recognition or semantic analysis on the speech segment, so that the speech to be recognized can be verified by speech recognition to obtain a first verification result, and then, according to the result type corresponding to the first verification result, the terminal device is controlled to perform a setting operation, and / or the speech to be recognized is subjected to a second verification process based on the first verification result. By adopting the method of the present invention, the speech to be recognized can be stored in the target storage area, thereby reducing the probability of speech interaction errors by performing a secondary recognition verification on the stored speech to be recognized. This solves the problem that speech recognition processing is prone to errors in the related art, resulting in low accuracy of speech recognition and low user experience.

[0035] In an exemplary embodiment, the speech processing result includes a speech recognition result and a semantic analysis result, and the method further includes: when the speech recognition result corresponding to the speech segment satisfies any one of the following conditions: the speech recognition result is empty, a boundary detection error occurs, and the speech recognition result fails to match, determining that the speech processing result corresponding to the speech segment meets the result abnormality condition, wherein the speech recognition result matching failure indicates that no recognition statement matching the speech recognition result is found in the first database; and / or when the semantic analysis result corresponds to the semantic analysis result matching failure, determining that the speech processing result corresponding to the speech segment meets the result abnormality condition, wherein the semantic analysis result is the result obtained by the semantic processing module performing semantic analysis processing on the speech recognition result, and the semantic analysis result matching failure indicates that no operation intention matching the semantic analysis result is found in the second database. In this embodiment, the recognition sentence matching the speech recognition result can be queried from the first database. If the recognition sentence matching the speech recognition result is not found in the first database, the speech processing result meets the abnormal result condition. For example, when the speech processing result corresponding to the speech segment is "kill mosquitoes", the recognition sentence matching the speech recognition result can be queried from the first database. If there is no recognition sentence matching "kill mosquitoes" in the first database, it can be determined that the speech recognition result meets the abnormal result condition, that is, the speech processing result is an abnormal result. In addition, the first database can be established in advance. For example, various words, sentences and other materials can be collected in advance and stored in the database to generate the first database. Sentences that have been recognized but do not exist in the first database can also be stored in the first database, and so on.

[0036] In the above embodiment, a semantic processing module can also be used to analyze the semantics of the voice segment included in the voice to be recognized, and then query the operation intention matching the semantic analysis result from the second database based on the semantic analysis result. If the operation intention matching the semantic analysis result is not found in the second database, the semantic analysis result meets the result abnormality condition. For example, when the semantic analysis result corresponding to the voice segment is "washing clothes", the operation intention matching the semantic analysis result can be queried in the second database. If there is no operation intention matching "washing clothes" in the second database, it can be determined that the semantic analysis result meets the abnormal result condition, that is, the semantic analysis result is an abnormal result. In addition, the second database can be established in advance. For example, various types of semantic operation intentions and other information can be collected in advance and stored in the database to generate the second database. The semantics that have been analyzed but do not exist in the second database can also be stored in the second database, and so on.

[0037] In an exemplary embodiment, the speech to be recognized is the original speech collected by the terminal device, the speech segment is part of the audio in the original speech, and the terminal device is controlled to perform a setting operation according to the result type corresponding to the first verification result, including: when it is determined that the result type represents that the speech to be recognized is a wake-up type speech, the terminal device is controlled to perform a wake-up operation and prohibit the first verification result from being sent to the semantic processing module; or, when it is determined that the result type represents that the speech to be recognized is a noise type audio, the terminal device is controlled to perform a sound pickup operation and prohibit the first verification result from being sent to the semantic processing module. In this embodiment, when the voice to be recognized is "Xiaobai" (it can also be "Xiaohei", "Xiaobu", etc.), and when the result type of the recognition verification of the voice to be recognized is a wake-up voice, the terminal device can be controlled to perform the operation of waking up "Xiaobai", and the wake-up voice is prohibited from being sent to the semantic processing module. In addition, there can be multiple terminal devices, so the priority of controlling the terminal device to perform the wake-up operation can be determined according to the distance from the multiple terminal devices. For example, when there are 3 terminal devices, among which the distance to the first terminal device is the closest, the distance to the second terminal device is the second closest, and the distance to the third terminal device is the farthest, the first terminal device with the closest distance can be preferentially controlled to perform the wake-up operation. In addition, the voice to be recognized may include multiple voice segments, and then when the voice recognition result and / or semantic analysis result of a voice segment included in the voice to be recognized is an abnormal result, other voice segments included in the voice to be recognized can be collected, and further voice recognition and / or semantic analysis can be performed on other voice segments. It should be noted that the above-mentioned example of the priority of the voice to be recognized and the above-mentioned terminal device is only an exemplary embodiment, and the priority of the voice to be recognized and the terminal device is not limited to the above-mentioned example.

[0038] In the above embodiment, when the result type of the recognition verification of the speech to be recognized is noise-type audio, the terminal device is controlled to perform a sound pickup operation, and the verification result is prohibited from being sent to the semantic processing module. For example, when the result type of the recognition verification of the speech to be recognized is construction noise, the terminal device is controlled to collect audio, and the construction noise is prohibited from being sent to the semantic processing module, thereby further improving the user experience. It should be noted that the above example of noise-type audio is only an exemplary embodiment, and noise-type audio is not limited to the above example.

[0039] In an exemplary embodiment, the second verification processing of the speech to be recognized based on the first verification result includes: when the result type of the first verification result indicates that the speech to be recognized is empty and / or a boundary detection error occurs, sending the first verification result and the speech to be recognized to an error diagnosis system; and using the error diagnosis system, performing the second verification processing on the original speech based on the first verification result and the speech to be recognized. In this embodiment, when the result type of the first verification result of the speech to be recognized is a boundary detection error, the first verification result and the speech to be recognized can be sent to an error diagnosis system. For example, when the speech to be recognized is "Xiaobai, I want to watch a science fiction TV series", the speech to be recognized is recognized and verified, and the result of the recognition verification is "I want to watch", that is, the result type is a boundary detection error. Then, the result type and the speech to be recognized can be sent to the error diagnosis system, and the error diagnosis system performs a second verification process on the original audio in the target storage area. Of course, the error diagnosis system can also be controlled to send the result type and the speech to be recognized to the background developer, and the developer can locate and analyze the problem and provide a quick and effective processing method. It should be noted that the above-mentioned example of boundary detection error is only an exemplary embodiment, and boundary detection error is not limited to the above-mentioned example. In an exemplary embodiment, the second verification process of the speech to be recognized based on the first verification result includes: when the result type of the first verification result indicates that the semantic recognition result of the speech to be recognized is normal, performing semantic analysis on the first verification result to obtain the semantic analysis result of the speech to be recognized; matching the semantic analysis result of the speech to be recognized with the operation intention in a second database; in case of a matching failure, sending the semantic analysis result of the speech to be recognized, the matching failure result and the speech to be recognized to an error diagnosis system; and using the error diagnosis system, performing the second verification process on the speech to be recognized based on the semantic analysis result of the speech to be recognized, the matching failure result and the speech to be recognized. In this embodiment, when the operation intention that matches the semantic analysis result of the voice to be recognized is not found in the second database, there is no need to perform in-depth analysis on the recognized voice at this time to avoid extending the operation time of responding to the voice processing result. Therefore, the semantic analysis result of the voice to be recognized, the matching failure result and the voice to be recognized can be sent to the error diagnosis system, and the error diagnosis system performs a second verification process on the voice to be recognized in the target storage area. Of course, the error diagnosis system is controlled to send the semantic analysis result of the voice to be recognized, the matching failure result and the voice to be recognized to the background developer, who locates and analyzes the problem and provides a quick and effective processing method.

[0040] In an exemplary embodiment, the method further includes: in the case of successful matching, obtaining an operation instruction corresponding to the operation intention, and controlling the terminal device to execute the operation instruction; controlling the terminal device to update the stored second speech recognition model based on the first speech recognition model stored by the server, wherein the server uses the first speech recognition model for speech recognition, and the terminal device uses the second speech recognition model for speech recognition. In this embodiment, the operation instruction can be a command instruction issued to a terminal device (for example, a smart TV, a smart air conditioner, a smart curtain, a smart alarm clock, etc.), wherein the command instruction can be a command to a smart TV to turn on the TV, a command to a smart air conditioner to turn on the air conditioner, a command to a smart curtain to open the curtain, etc., or a command to a smart air conditioner to increase the temperature, a command to a smart air conditioner to lower the temperature, or a command to a smart alarm clock to set a timer, a command to a smart alarm clock to set an alarm, and so on.

[0041] In the above embodiment, both the terminal device and the server can be configured with a speech recognition model, wherein the speech recognition model can be used to recognize speech, including recognizing the complete speech, recognizing partial fragments included in the speech, etc. The terminal device can use the configured first speech recognition model to perform speech recognition on the original speech. When the speech recognition result is an abnormal result, the original speech is sent to the server. The server uses the second speech recognition model to recognize and verify the original speech, and instructs the terminal device to update the speech recognition result recognized by the first speech recognition model with the correct speech recognition result recognized and verified by the second speech recognition model. In addition, when the terminal device cannot recognize the original speech, the original speech can be sent to the server, and the original audio is recognized by the server, and the recognized correct speech recognition result is sent to the terminal for storage, thereby further improving the accuracy of speech recognition verification.

[0042] In an exemplary embodiment, the method further includes: the method further includes: in the case where no query operation for the voice to be recognized in the target storage area is received within a predetermined time period, deleting the voice to be recognized stored in the target storage area. In this embodiment, the predetermined time period is a value that can be preset, and can be set to 30 minutes, 1 hour, 6 hours, one day, three days, one week, etc. For example, when the predetermined time period is three days, in the case where no query operation for the voice to be recognized in the target storage area is received within three days, the voice stored in the target storage area received within the three days is deleted. For example, within three days, the voices uttered by the user or operator are all recognized normally. In the case of no abnormal results, all the voices received within the three days stored in the target storage area can be deleted to reduce the storage pressure of the target storage area.

[0043] Obviously, the above described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. The present invention will be specifically described below in conjunction with the embodiments.

[0044] Figure 3 FIG. 1 is a general flow chart of a speech recognition verification system according to an embodiment of the present invention. Figure 3 The process includes the following steps:

[0045] S302, start;

[0046] S304, recording original audio;

[0047] S306, sending to ASR (Automatic Speech Recognition, automatic speech recognition technology) end for recognition;

[0048] Send the recorded raw audio to ASR (Automatic Speech Recognition).

[0049] S308, original audio backup;

[0050] The recorded original audio is sent to a server (corresponding to the target storage area of ​​the above server) for storage and backup.

[0051] S310, the ASR result is sent to the NLP (Natural Language Processing) end;

[0052] S312, performing a first judgment to determine whether the NLP result match is correct, and if the first judgment result is no, sending the wrong NLP matching result to the original audio backup end of step S308;

[0053] When NLP does not have a matching result, it can be determined that the ASR recognition result may be abnormal. At this time, it is necessary to find the original audio at the corresponding time (corresponding to the above-mentioned voice to be recognized) at the original audio backup end for comparison and verification.

[0054] S314, when the first judgment result is yes, sending the correct NLP matching result to the terminal device;

[0055] Under normal circumstances, after the ASR end recognizes the result, it sends the recognition result to the NLP (natural speech processing) end, and the NLP matches the result and sends it to the terminal device to respond to the user's voice request.

[0056] S316, end;

[0057] S318, performing a second determination to determine whether a query event is received within a set time period;

[0058] S320, if the second judgment result is no, delete the original audio before that moment;

[0059] The original audio backup end has a self-refresh mechanism. When no query signal is received within a certain period of time, the original audio within this period of time is deleted to reduce the background storage pressure.

[0060] S322, secondary verification;

[0061] The verification content may include whether the audio data is empty, whether it is a wake-up word, whether it is environmental noise, whether it is a vad (voice boundary detection) error, etc. When the ASR recognition result is empty or a vad error (for example, an incomplete sentence, etc.), the original audio data with a longer time near the corresponding moment is intercepted for analysis to reduce the problem of ASR parsed data being empty or incomplete sentences caused by terminal vad errors. When the ASR recognition result is a wake-up word or environmental noise, it is prohibited to send the ASR recognition result to the NLP end to avoid NLP responding to voices that are not intended by the user.

[0062] S324, performing a third judgment to determine whether the result of the NLP secondary matching is correct. If the result of the third judgment is yes, executing step S320 to delete the original audio before this moment (corresponding to the to-be-recognized speech stored in the above-mentioned target storage area);

[0063] S326, when the third judgment result is no, sending the speech to be recognized to the error diagnosis system (corresponding to the target system);

[0064] When the ASR recognition result of the secondary verification is still empty or has a vad error, the original audio within the corresponding period of time will be sent to the error diagnosis system, and the relevant developers will perform error diagnosis later. When the ASR recognition result of the secondary verification still has no matching result on the NLP side, the original audio within the corresponding period of time will be sent to the error diagnosis system, and the relevant developers will perform error diagnosis later.

[0065] S328, the developer diagnoses and executes step S316.

[0066] It can be seen from the above embodiments that when uploading the ASR recognition results to the server, the terminal will also send the original audio data to the server for storage. When the ASR recognition error occurs, the processing module in the server (for example, the error correction module) can quickly find the original audio at the corresponding time, perform verification, and determine and process whether the audio data is empty, whether it is a wake-up word, whether it is environmental noise, whether it is a vad error, etc. When the ASR recognition result is empty or a vad error, the original audio data with a longer time near the corresponding time (corresponding to the above-mentioned target voice) can be intercepted for analysis to reduce the problem of the ASR recognition result being empty or the ASR parsed data being empty or the sentence being incomplete due to a vad error. When the ASR recognition result is a wake-up word or environmental noise, it is prohibited to send the ASR recognition result to the NLP end to avoid the NLP responding to non-operator or user intentions, which greatly improves the accuracy of speech recognition and further enhances the user experience of the operator or user.

[0067] It should also be noted that if there are still abnormal results after the verification, the original audio within a period of time at a certain moment can be sent to the error diagnosis system, and the relevant developers will subsequently perform error diagnosis. In addition, when the verification results sent to NLP still cannot match the NLP results, the original audio within a period of time at the corresponding moment can also be sent to the error diagnosis system, and the relevant developers will subsequently perform error diagnosis, which reduces the probability of NLP making a fallback response when there is no matching result. Subsequently, it reduces the time for relevant developers to locate and analyze problems to a certain extent, and improves the efficiency of speech recognition error correction.

[0068] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0069] In this embodiment, a speech processing device is also provided. Figure 4 is a structural block diagram of a speech processing device according to an embodiment of the present invention. Figure 4 As shown, the device comprises:

[0070] The acquisition module 42 is used to acquire the to-be-recognized speech from the target storage area of ​​the server when the speech processing result corresponding to the speech segment collected by the terminal device meets the result abnormality condition, wherein the to-be-recognized speech includes the speech segment, and the speech processing result includes the result obtained by performing speech recognition or semantic analysis on the speech segment;

[0071] The recognition module 44 is used to perform speech recognition verification based on the speech to be recognized to obtain a first verification result;

[0072] The processing module 46 is used to control the terminal device to execute a setting operation according to the result type corresponding to the first verification result, and / or perform a second verification process on the speech to be recognized based on the first verification result.

[0073] In an optional embodiment, the acquisition module 42 includes: a first determination unit, for determining that the speech processing result corresponding to the speech segment satisfies an abnormal result condition when the speech recognition result corresponding to the speech segment satisfies any one of the following conditions: the speech recognition result is empty, a boundary detection error occurs, and the speech recognition result fails to match, wherein the speech recognition result matching failure indicates that no recognition statement matching the speech recognition result is found in the first database; a second determination unit, for determining that the speech processing result corresponding to the speech segment satisfies an abnormal result condition and / or when the semantic analysis result corresponds to the semantic analysis result matching failure, wherein the semantic analysis result is a result obtained by the semantic processing module performing semantic analysis processing on the speech recognition result, and the semantic analysis result matching failure indicates that no operation intention matching the semantic analysis result is found in the second database.

[0074] In an optional embodiment, the above-mentioned processing module 46 includes: a first control unit, which is used to control the terminal device to perform a wake-up operation and prohibit the first verification result from being sent to the semantic processing module when it is determined that the result type represents that the voice to be recognized is a wake-up type voice; and a second control unit, which is used to control the terminal device to perform a sound pickup operation and prohibit the first verification result from being sent to the semantic processing module when it is determined that the result type represents that the voice to be recognized is a noise type audio.

[0075] In an optional embodiment, the processing module 46 further includes: a first sending unit, for sending the first verification result and the speech to be recognized to an error diagnosis system when the result type of the first verification result indicates that the speech to be recognized is empty and / or a boundary detection error occurs; and a first execution unit, for using the error diagnosis system to perform the second verification processing on the original speech based on the first verification result and the speech to be recognized.

[0076] In an optional embodiment, the processing module 46 further includes: an analysis unit, which is used to perform semantic analysis on the first verification result to obtain a semantic analysis result of the speech to be recognized when the result type of the first verification result indicates that the semantic recognition result of the speech to be recognized is normal; a matching unit, which is used to match the semantic analysis result of the speech to be recognized with the operation intention in the second database; a second sending unit, which is used to send the semantic analysis result of the speech to be recognized, the matching failure result and the speech to be recognized to the error diagnosis system when the matching fails; and a second execution unit, which is used to use the error diagnosis system to perform the second verification processing on the speech to be recognized based on the semantic analysis result of the speech to be recognized, the matching failure result and the speech to be recognized.

[0077] In an optional embodiment, the above-mentioned device also includes: a first control module, which is used to obtain the operation instruction corresponding to the operation intention when the match is successful, and control the terminal device to execute the operation instruction; a second control module, which is used to control the terminal device to update the stored second speech recognition model based on the first speech recognition model stored in the server, wherein the server uses the first speech recognition model for speech recognition, and the terminal device uses the second speech recognition model for speech recognition.

[0078] In an optional embodiment, the above-mentioned device further includes: a deleting module, which is used to delete the speech to be recognized stored in the target storage area when no query operation on the speech to be recognized in the target storage area is received within a predetermined time period.

[0079] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0080] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0081] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0082] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0083] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0084] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0085] Through the speech processing method provided by the present invention, when the first speech recognition verification is erroneous, the original audio can be analyzed and recognized again in connection with the context of the original audio, which on the one hand improves the accuracy of speech recognition, and on the other hand reduces the time for relevant developers to locate and analyze problems, further improving the user experience.

[0086] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0087] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for speech processing, characterized in that: include: In the case where the speech processing result corresponding to the speech segment collected by the terminal device meets the result abnormality condition, the speech to be recognized is obtained from the target storage area of ​​the server, wherein the speech to be recognized includes the speech segment, the speech to be recognized is the original speech collected by the terminal device, and the speech segment is part of the audio in the original speech, and in the case where the speech processing result includes the result obtained by performing speech recognition on the speech segment, and the speech recognition result corresponding to the speech segment meets any one of the following conditions: the speech recognition result is empty, a boundary detection error occurs, and the speech recognition result fails to match, it is determined that the speech processing result corresponding to the speech segment meets the result abnormality condition, and the speech recognition result fails to match, indicating that no recognition sentence matching the speech recognition result is found in the first database; Performing speech recognition verification based on the speech to be recognized to obtain a first verification result; Determining a result type corresponding to the first verification result; In the case where it is determined that the result type indicates that the speech to be recognized is a wake-up type speech, controlling the terminal device to perform a wake-up operation and prohibiting the first verification result from being sent to the semantic processing module; In the case where it is determined that the result type indicates that the to-be-recognized speech is noise-type audio, controlling the terminal device to perform a sound pickup operation, and prohibiting the first verification result from being sent to the semantic processing module; In the case where the result type of the first verification result indicates that the speech to be recognized is empty and / or a boundary detection error occurs, the first verification result and the speech to be recognized are sent to an error diagnosis system; and using the error diagnosis system, a second verification process is performed on the original speech based on the first verification result and the speech to be recognized.

2. The method according to claim 1, characterized in that In the case where the speech processing result includes a result obtained by performing semantic analysis on the speech segment, and the semantic analysis result fails to match the corresponding semantic analysis result, it is determined that the speech processing result corresponding to the speech segment meets the result abnormality condition, wherein the semantic analysis result is a result obtained by performing semantic analysis processing on the speech recognition result by a semantic processing module, and the semantic analysis result matching failure indicates that no operation intention matching the semantic analysis result is found in the second database.

3. The method according to claim 2, characterized in that After determining the result type corresponding to the first verification result, the method further includes: When the result type of the first verification result indicates that the semantic recognition result of the speech to be recognized is normal, performing semantic analysis on the first verification result to obtain a semantic analysis result of the speech to be recognized; Matching the semantic analysis result of the speech to be recognized with the operation intention in the second database; In case of a matching failure, the semantic analysis result of the speech to be recognized, the matching failure result and the speech to be recognized are sent to an error diagnosis system; The error diagnosis system is used to perform the second verification process on the speech to be recognized based on the semantic analysis result of the speech to be recognized, the matching failure result, and the speech to be recognized.

4. The method according to claim 3, characterized in that The method further comprises: If the match is successful, obtaining the operation instruction corresponding to the operation intention, and controlling the terminal device to execute the operation instruction; Control the terminal device to update the stored second speech recognition model based on the first speech recognition model stored in the server, wherein the server uses the first speech recognition model to perform speech recognition, and the terminal device uses the second speech recognition model to perform speech recognition.

5. The method according to claim 1, characterized in that The method further comprises: When no query operation on the speech to be recognized in the target storage area is received within a predetermined period of time, the speech to be recognized stored in the target storage area is deleted.

6. A speech processing device, characterized in that: include: an acquisition module, configured to acquire the to-be-recognized speech from a target storage area of ​​a server when a speech processing result corresponding to a speech segment collected by a terminal device satisfies a result abnormality condition, wherein the to-be-recognized speech includes the speech segment, the to-be-recognized speech is the original speech collected by the terminal device, and the speech segment is a portion of the audio in the original speech, and when the speech processing result includes a result obtained by performing speech recognition or semantic analysis on the speech segment, and the speech recognition result corresponding to the speech segment satisfies any one of the following conditions: the speech recognition result is empty, a boundary detection error occurs, and a speech recognition result matching failure occurs, it is determined that the speech processing result corresponding to the speech segment satisfies the result abnormality condition, and the speech recognition result matching failure indicates that a recognition sentence matching the speech recognition result is not found in a first database; A recognition module, configured to perform speech recognition verification based on the speech to be recognized, and obtain a first verification result; A processing module, used to determine a result type corresponding to the first verification result; The processing module includes: a first control unit, which is used to control the terminal device to perform a wake-up operation and prohibit the first verification result from being sent to the semantic processing module when it is determined that the result type indicates that the speech to be recognized is a wake-up type speech; A second control unit is used to control the terminal device to perform a sound pickup operation and prohibit sending the first verification result to the semantic processing module when it is determined that the result type indicates that the speech to be recognized is noise-type audio; A first sending unit is used to send the first verification result and the speech to be recognized to an error diagnosis system when the result type of the first verification result indicates that the speech to be recognized is empty and / or a boundary detection error occurs; and a first execution unit is used to use the error diagnosis system to perform a second verification process on the original speech based on the first verification result and the speech to be recognized.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method according to any one of claims 1 to 5 when executed.

8. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 5 through the computer program.

Images