Voice processing method, apparatus and system

Abstract

A speech processing method, device and system are disclosed. The speech processing method comprises: obtaining speech information; inputting the obtained speech information into a cloud speech processing engine and a local speech processing engine; determining a speech processing result to be used according to an offline-online arbitration rule; and performing a corresponding operation based on the determined speech processing result. The application inputs a speech stream into a cloud engine and a local engine at the same time, and selects a proper online or offline processing result through a reasonably set offline-online arbitration strategy. Therefore, a dynamic real-time decision for offline-online processing result selection can be realized, and is not affected by a weak network environment and a connection condition of an application and a cloud.

Description

[0001] The present application relates to the field of information processing, and in particular to a voice processing method, device and distributed system. BACKGROUND

[0002] Generally, the voice interaction device of the terminal can upload the received voice to the cloud to perform voice recognition and natural language understanding by means of the powerful processing capability of the cloud. Alternatively, in the case of poor networking conditions or simple application scenarios, the voice interaction device can also complete the entire voice interaction link locally. Therefore, how to appropriately select online (cloud) or offline (local) processing results for operation has become a problem to be solved in the field. SUMMARY

[0003] One of the technical problems to be solved by the present disclosure is to provide a voice processing scheme, which simultaneously inputs a voice stream into a cloud engine and a local engine, and selects appropriate online or offline processing results through a reasonably set online-offline arbitration strategy. For this purpose, dynamic real-time decision for online-offline processing result selection can be realized, and is not affected by weak network environment and application and cloud connection conditions.

[0004] According to a first aspect of the present disclosure, a voice processing method is provided, comprising: obtaining voice information; inputting the obtained voice information into a cloud voice processing engine and a local voice processing engine; determining a voice processing result to be used according to an online-offline arbitration rule; and performing a corresponding operation based on the determined voice processing result.

[0005] According to a second aspect of the present disclosure, a voice processing device is provided, comprising: a voice information obtaining unit configured to obtain voice information; a local voice processing engine configured to perform local voice processing on the obtained voice information; a communication unit configured to input the obtained voice information into a cloud voice processing engine; an arbitration unit configured to determine a voice processing result to be used according to an online-offline arbitration rule; and an operation unit configured to perform a corresponding operation based on the determined voice processing result.

[0006] According to a third aspect of the present disclosure, a voice processing system is provided, comprising: a voice processing device configured to perform the voice processing method according to the first aspect of the present application; and a server comprising the cloud voice processing engine and configured to issue a cloud voice processing result based on the obtained voice information.

[0007] According to a fourth aspect of the present disclosure, a computing device is provided, comprising: a processor; and a memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method according to the first aspect described above.

[0008] According to a fifth aspect of the present disclosure, there is provided a non-transitory machine readable storage medium having stored thereon executable code that, when executed by a processor of an electronic device, causes the processor to perform the method of the first aspect described above.

[0009] Thus, the present application provides a set of online and offline voice processing result selection scheme. The scheme simultaneously performs local and cloud voice processing procedures, and makes real-time decisions according to the timeliness and confidence of the processing results, which can always provide timely and accurate voice processing results for users. BRIEF DESCRIPTION OF DRAWINGS

[0010] The above and other objects, features and advantages of the present disclosure will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which like reference characters refer to like parts throughout the figures, in which:

[0011] Figure 1 A schematic flow chart of a voice processing method according to an embodiment of the present application is shown.

[0012] Figure 2 A schematic diagram of a voice interaction link processing procedure is shown.

[0013] Figure 3 A flow execution example of a voice processing method according to the present application is shown.

[0014] Figure 4 An example of online and offline arbitration rules according to the present application is shown.

[0015] Figure 5 An example of simultaneously performing cloud and local procedures in a weak network is shown.

[0016] Figure 6 A schematic diagram of the composition of a voice processing device according to an embodiment of the present application is shown.

[0017] Figure 7 A schematic diagram of the composition of a voice processing system capable of implementing the present application is shown.

[0018] Figure 8 A structural schematic diagram of a computing device according to an embodiment of the present application, which can be used to implement the voice processing method described above, is shown. DETAILED DESCRIPTION

[0019] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0020] The voice interaction device of the terminal can upload the received voice to the cloud for voice recognition and natural language understanding by virtue of the powerful processing capability of the cloud. For a home scenario, the network service is stable and low in cost, and the cloud processing for voice interaction can be guaranteed. However, for a mobile (for example, a vehicle-mounted) scenario, a mobile network such as a mobile phone traffic is usually used to provide access. If a tunnel or a basement is entered, a weak network or no network may occur, resulting in unstable cloud connection. When the cloud service is unreliable, a local voice processing service that does not perform network connection and does not rely on the processing capability of the cloud needs to be used. How to properly select online (cloud) or offline (local) processing results for operation has become a problem to be solved in the field.

[0021] In some solutions, a ping (Packet Internet Groper, Internet packet explorer) method is used to select whether to use the processing capability of the cloud. Specifically, the ping can be used to determine whether a local host can successfully exchange (send and receive) data packets with another host, and according to the returned information, it can be inferred whether the network is smooth. When the ping method detects that the network is not smooth, the local processing capability can be switched to. However, the ping method can only check the network path and cannot judge the weak network with slow feedback, nor can it judge the connection of the application.

[0022] In some other solutions, a configuration file can be used to select the cloud and local processing. However, the file configured in advance is not flexible enough and cannot cover various special scenarios and does not have the ability to respond flexibly.

[0023] Therefore, the present application provides a voice processing scheme, which simultaneously inputs a voice stream into a cloud engine and a local engine, and selects appropriate online or offline processing results through a reasonably set online-offline arbitration strategy. Therefore, dynamic real-time decision for online-offline processing result selection can be realized, and is not affected by the weak network environment and the connection of the application and the cloud.

[0024] Figure 1 A schematic flowchart of a voice processing method according to an embodiment of the present application is shown. The voice method described above can be executed by any device or application installed thereon having both cloud access capability and local voice processing capability, and is used to give the correct operation required by the user based on the obtained voice information.

[0025] At step S110, the voice information is acquired. Here, the local device can acquire voice information directly through voice collection via a microphone arranged therein or externally connected. In a voice collection network composed of smart voice stickers or smart voice sockets, the local device can acquire voice information collected by other voice collection points via short-range wireless communication.

[0026] At step S120, the acquired voice information is input into the cloud voice processing engine and the local voice processing engine. In some embodiments, the collected or acquired voice information can be directly sent to the local and cloud engines. In other embodiments, the acquired voice information can also be processed in the front end (for example, noise reduction and MFCC feature extraction), and the processed voice information is sent to the cloud and local engines.

[0027] For ease of understanding, Figure 2 A processing flow diagram of the voice interaction link is shown. The voice interaction link refers to the process of implementing voice interaction involving audio data collection, front-end signal processing, wake-up response, voice recognition, voiceprint recognition, natural language processing, and voice synthesis unit modules. These unit modules work together to complete the voice interaction function. In different application scenarios, some or all of the above-mentioned interaction link modules can be involved. For example, in the case of key wake-up, the wake-up response unit based on the wake-up word is not necessary. In addition, the voiceprint recognition and voice synthesis functions are not necessary for some application scenarios. Figure 2 Only the most core unit modules are shown in the figure. In other implementations, the interaction link can also include functional modules such as the wake-up response and voiceprint recognition described above.

[0028] As Figure 2As shown, the audio collected user voice is processed by the speech recognition module to generate a speech recognition result, i.e., a text instruction corresponding to the user utterance. Speech recognition, also known as Automatic Speech Recognition (ASR), is a technology that converts speech (e.g., user input speech) into text. For example, via ASR, the user input speech "turn on the air conditioner" can be converted into the literal meaning "turn on the air conditioner". Subsequently, the Natural Language Understanding (NLU) module performs semantic analysis on the user utterance. Here, natural language understanding refers to an algorithm / system that recognizes the meaning of text. In a speech processing system, NLU can recognize the user voice instruction as a specific domain intent. In the present disclosure, the NLU module can also be referred to as an intent understanding module. A domain refers to a certain specific problem in the field of natural language understanding, such as weather, time, etc., and an intent refers to a specific problem in a certain domain in the field of natural language understanding, such as querying the weather, querying the time, setting an alarm, etc. After understanding the domain and intent in the user voice instruction, the above domain and intent can be fed into an intent execution model block, which can select the system behavior to be executed according to the NLU semantic analysis result (i.e., a specific domain and intent). For example, based on the text "turn on the air conditioner", the NLU module can understand that the semantic intent is to "start" the "built-in-vehicle air conditioner". In order to generate voice feedback, the speech link can also include a natural language generation module (NLG) to generate natural language or system utterance according to the system behavior result. Finally, the generated language can be read to the user by the Text-to-Speech (TTS) module.

[0029] Here, the cloud-side speech processing engine and the local-side speech processing engine each at least need to have the speech recognition and natural language understanding capabilities mentioned above in the speech link. The speech processing engine equipped with the ASR and NLU modules can recognize user voice and understand the intent behind the semantics, thereby enabling subsequent operations to be performed. For example, when the user gives the voice input "turn on the air conditioner" as described above, the speech processing engine can first recognize the text "turn on the air conditioner" via ASR, and understand the semantic intent of the text "turn on the air conditioner" via NLU, which is to "start" the "built-in-vehicle air conditioner". At this time, the speech processing result can be to generate an air conditioner on instruction, and the device (e.g., the smart car here) can then execute the operation of turning on the air conditioner based on the instruction. Optionally, the speech processing engine can also generate a feedback voice instruction for feedback based on the NLG module. When the vehicle air conditioner is turned on, the TTS module can generate a voice feedback of "the air conditioner has been turned on" based on the feedback voice instruction.

[0030] At step S130, a voice processing result to be used is determined according to an off-online arbitration rule. Here, the off-online arbitration rule and the specific involved parameter size can be pre-set by the system, or can be customized by the user on the setting page. The off-online arbitration rule is a rule for determining in which cases the local processing result generated by the local voice processing engine is selected, and in which cases the cloud processing result generated by the cloud voice processing engine is selected.

[0031] Subsequently, a corresponding operation is performed based on the determined voice processing result. As described above, the voice processing result can be feedback given after the input voice intention is understood. The feedback can be performing a specific operation, for example, turning on the air conditioner, or can be giving a corresponding voice feedback, for example, reporting "the air conditioner has been turned on".

[0032] From this, inputting the acquired voice information into the cloud voice processing engine and the local voice processing engine can include sending the voice information to the cloud voice processing engine while sending the voice information to the local voice engine. Figure 3 A flow execution example of the voice processing method according to the present application is shown.

[0033] As shown in the figure, when the user wakes up the voice assistant, the voice stream can be input to the cloud engine (through the network) and the local engine at the same time, and then the off-online arbitration rule selects a certain processing result for execution according to the return of the local and cloud processing results. Thus, by always performing cloud and local end-cloud simultaneous streaming, and selecting a more appropriate (usually faster and / or more accurate) processing result based on the rule, dynamic real-time decision making can be made according to the properties of the processing result, without being affected by the weak network environment and the connection status of the application and the cloud.

[0034] As described above, the voice information used for cloud and local simultaneous processing can be a voice command given after the user wakes up the voice assistant. Therefore, the step S110 of acquiring voice information includes acquiring voice information received after the voice assistant is woken up. In other words, before step S110, there is also a step of listening to the wake-up word and being woken up.

[0035] In some embodiments, the same device can include different voice assistants. For example, in a vehicle scenario, the manufacturer of the vehicle will usually include a voice assistant function in the equipped vehicle system. The user can wake up the native voice assistant of the vehicle by, for example, “Hello XX (vehicle name or nickname of the vehicle)”. The native voice assistant of the vehicle usually has good vehicle control capability. However, compared with some specialized voice processing platforms, the vehicle native assistant and the cloud processing capability it relies on have weaker intent understanding capability in many other scenarios (for example, navigation, music appreciation, free conversation). Therefore, a specialized voice processing platform can also be additionally accessed. The access of the above platform can be realized through different wake-up words. For example, in addition to being able to wake up the native voice assistant of the vehicle by saying, for example, “Hello XX (vehicle name or nickname of the vehicle)”, the user can also wake up the external voice assistant with stronger voice processing capability by saying “XXXX (external voice assistant name)”.

[0036] At this time, the voice processing method of the present application can also include: waking up different voice assistants based on different wake-up words. For a hardware low-power wake-up module equipped on the vehicle, the recognition of multiple wake-up words by the wake-up module can be realized through a software algorithm. Different voice assistants usually correspond to different cloud processing platforms. Therefore, inputting the obtained voice information into the cloud voice processing engine and the local voice processing engine includes: inputting the obtained voice information into the same local voice processing engine and the cloud voice processing engine corresponding to the woken-up voice assistant. In other words, different voice assistants can use the same local voice processing engine, which usually corresponds to the same voice processing module; and can each connect a different server (i.e., a different cloud processing platform) via the communication unit. The user can select the appropriate voice assistant for interaction by saying the corresponding wake-up word in the presence of different needs.

[0037] The arbitration rules and preferred embodiments thereof will be described below in connection with Figure 4 , and can be considered as a refinement of Figure 1 step S130.

[0038] Figure 4 An example of the off-line / on-line arbitration rule according to the present application is shown. As described before, the off-line / on-line arbitration rule of the present application is a rule for determining in which cases the local processing result (off-line result) generated using the local voice processing engine is selected, and in which cases the cloud processing result (on-line result) generated using the cloud voice processing engine is selected. Since there is a need for timeliness of feedback for voice, the off-line / on-line arbitration rule of the present application needs to consider the requirement of timeliness while ensuring accuracy.

[0039] Since the processing capability of the cloud model is stronger than the processing capability on the terminal, asFigure 4 In the case that the local processing result is returned first, the cloud processing result is preferably used as the voice processing result to be used. Further, to avoid waste of computing power, the current processing flow of the local voice processing engine for the voice information can be cancelled at the same time.

[0040] Further, in the case that the local processing result is returned first, the voice processing result to be used needs to be determined according to the confidence of the local processing result. For example, in the case that the local processing result has a relatively high confidence (e.g., the judgment probability of the local model for the current result is high), the local processing result can be deemed to be reliable and directly used. In the case that the confidence of the local processing result is limited, the cloud processing result needs to be selected in combination with the timeliness of the return of the cloud processing result.

[0041] Specifically, when the confidence of the local processing result is higher than a first threshold (e.g., the local result accuracy is greater than 90% as shown in the figure), the local processing result can be deemed to be reliable, and the local processing result is used as the voice processing result to be used. Accordingly, to avoid waste of computing power, the current processing flow of the cloud voice processing engine for the voice information can be cancelled at the same time, for example, by sending a cancellation request to the server.

[0042] When the confidence of the local processing result is lower than the first threshold and higher than a second threshold (e.g., the local result accuracy is between 60% and 90% as shown in the figure), the local processing result can be deemed to be less reliable, and the cloud result is still preferably used, but the waiting time of the cloud result needs to meet the timeliness requirement. Therefore, in the case that the local processing result is returned first, determining the voice processing result to be used according to the confidence of the local processing result can include: when the confidence of the local processing result is lower than the first threshold and higher than the second threshold, continuing to wait for the cloud processing result to a first predetermined time (e.g., 1.5 seconds as shown in the figure); if the cloud processing result is returned within the first predetermined time, using the cloud processing result as the voice processing result to be used; and if the cloud processing result is not returned within the first predetermined time, using the local processing result as the voice processing result to be used. In this way, while the accuracy is taken into account, the timeliness requirement for feedback of the voice is also considered. Similarly, the selection of a processing result can trigger the cancellation of the flow of another engine at the same time.

[0043] When the confidence of the local processing result is lower than a second threshold (e.g. 60% as shown in the figure), then the local result can be considered as not reliable, in which case the cloud processing result needs to be waited for and the local result can be discarded accordingly. Further, for timeliness consideration, the second predetermined time for waiting for the cloud processing result when the local processing result is not reliable can be longer (or at least not shorter) than the first predetermined time for waiting for the cloud processing result when the local processing result is less reliable. For example, when the first predetermined time is 1.5 seconds, the second predetermined time can be a user-acceptable waiting time limit, e.g. 3 seconds. If the cloud processing result is returned within the second predetermined time, then the cloud processing result is used as the speech processing result to be used. If the cloud processing result is not returned within the second predetermined time, then a timeout processing is triggered.

[0044] The speech processing method as described above in connection with Figure 1 、 3 -4 is suitable for handling the case where there is network connection but the network connection is not good, e.g. in a weak network environment. Further, in addition to the local or cloud processing result selection based on the on-line or off-line arbitration rule as described above, the speech processing method of the present application can also improve the ability to obtain the cloud processing result in a weak network environment based on the judgment of the weak network environment.

[0045] As mentioned above, the local processing engine can include a local speech recognition module and a local natural language processing module. In order to reduce the uplink data volume, the speech processing method of the present application can further include: under a predetermined condition, uploading the local speech recognition result generated by the local speech recognition module as a substitute for the speech information to the cloud speech processing engine. This is because the amount of text data recognized by ASR is usually smaller than the amount of data required for directly uploading the speech stream.

[0046] Figure 5 An example of simultaneous cloud and local processing in a weak network environment is shown. As shown in the figure, in the acquisition and uploading phase, the speech acquisition module of the local device (and, in some embodiments, other speech acquisition devices in communication with the local device) monitors the speech command input from the user or the smart device, and the local device can perform preliminary processing on the acquired speech, e.g. noise reduction and feature extraction. Unlike Figure 3The voice stream is directly input to the local engine and the cloud engine at the same time. At this time, since it is determined that the network condition is poor, only the local ASR picking can be performed, and the picked voice command is transmitted to the cloud and local NLU modules. In the parallel processing stage of the cloud and the local, the server can perform subsequent processing on the picked voice command, such as NIP (natural speech processing) and NIU (natural speech understanding), and according to the processing result, command analysis and TTS output are performed. Correspondingly, the local module can also perform subsequent processing on the picked voice command, such as NIP (natural speech processing) and NIU (natural speech understanding), and according to the processing result, command analysis and TTS output are performed.

[0047] In the local processing stage, the processing results generated by the cloud and the local respectively (for example, each can include command analysis and / or TTS output) are also executed online-offline arbitration strategy. At this time, since the reliability of the cloud result is still higher than that of the local result, the arbitration strategy of giving priority to the cloud result and considering timeliness can also be performed. But since the online result is not calculated by the cloud through the whole link, the parameters in the arbitration strategy at this time can be different from Figure 4 the parameters shown.

[0048] Subsequently, the result obtained by arbitration can be used for local execution. Specifically, the analyzed command can be transmitted to the local device for execution, or can be executed by the target device under the control of the local device, for example. In addition, when there is audio output, the voice device can perform voice output through the built-in loudspeaker or the externally connected Bluetooth or traditional sound box.

[0049] Therefore, in the case of detecting a weak network state, by directly uploading the text recognized by the local ASR to the cloud instead of the original voice stream, the uplink data amount can be reduced, thereby better realizing voice recognition under a weak network condition. In some embodiments, in addition to uploading the recognition result with the highest probability of local ASR, N-Best (i.e., multiple preferred recognition results) can also be uploaded to facilitate the cloud to obtain more accurate NLU results according to more ASR data.

[0050] Alternatively or additionally, the uploaded data can also be compressed in the case of a weak network or a suspected weak network, for example, the voice stream is compressed by the speex-ogg algorithm.

[0051] Accordingly, in order to reduce the data of downlink transmission, the present application can further comprise: under the condition that the predetermined condition is met, causing the cloud voice processing engine to issue a cloud processing result with a minimum data amount. As mentioned above, the cloud voice processing engine has powerful voice processing functions. For the same voice input, different versions of processing results can be given in combination with the user's historical preferences, the current application scenario, etc. For example, when the user asks "what is the weather today?", a concise broadcast content can be given, such as "today is sunny, temperature 12-18°, east wind 2 levels", or a broadcast content containing more information can be given, such as "today is sunny, temperature 12-18°, east wind 2 levels, 6° lower than the previous day, please pay attention to add clothes". In the case of weak network, the local device can send an indication of poor network to the cloud, so that the cloud voice processing engine issues a cloud processing result with a minimum data amount to facilitate the local device to receive. After determining that the network is restored, the local device can send an indication of canceling the poor network to the cloud, so that the cloud voice processing engine gives a cloud processing version most suitable for the current scenario and the user.

[0052] The operation of reducing the uplink and / or downlink transmission data amount as described above can be performed under the condition that the predetermined condition is met. In other words, the current weak network condition can be inferred under the condition that the predetermined condition is met, and the corresponding operation of reducing the uplink and / or downlink transmission data amount is performed accordingly.

[0053] In one embodiment, the predetermined condition can include that no cloud processing result is received within a third predetermined time. The third predetermined time can be a predetermined time independent of the first and second predetermined times, or can be related to the first or second predetermined time. The third predetermined time can be determined according to the regular waiting time when the network is normal. For example, the regular waiting time for returning information from the cloud after uploading data to the cloud is 3 seconds, and if no returned processing result is received within 3 seconds, it can be considered that the current network condition is not good. At this time, the third predetermined time can be set to 3 seconds. In other cases, the third predetermined time can be appropriately extended, for example, set to 4 seconds.

[0054] In addition, the operation of reducing the uplink and / or downlink data amount as described above can also be performed only when the confidence of the local processing result is not high, so it is more necessary to obtain a cloud processing result with higher confidence. To this end, the predetermined condition can include that the confidence of the local processing result is lower than a third threshold. The third threshold can be independent of the first and second thresholds, or can be related to the first or second threshold. For example, the third threshold can be set to be the same as the second threshold, i.e. 60%.

[0055] In some cases, the operation of reducing the uplink and / or downlink data transmission amount can be performed for the current voice processing, or for subsequent voice processing.

[0056] For example, a user of a smart vehicle might input the voice command "open the sunroof" while the vehicle is in motion. While the local ASR (Automatic Sound Recognition) accurately recognizes the spoken text, the local NLU (Natural Language Logic) cannot recognize the "ba" construction, resulting in a low confidence level for the local processing result (e.g., below 60%). If the user is passing through a road with poor signal, they may not receive cloud processing feedback after the usual 3-second wait (e.g., because the voice stream upload is not yet complete). At this point, the local device can initiate a new request and directly send the locally recognized ASR text to the cloud. Because the ASR data size is smaller, it is uploaded first, and the signal instructing the sunroof to open is successfully transmitted. Five seconds after the user's voice input, the sunroof begins to open. Thus, through a secondary upload of the current voice data, although the user wait time is longer, the desired operation is still completed within a tolerable timeframe.

[0057] Similarly, in the example of the user's voice input "open the sunroof," if the user waits 3 seconds and then re-enters "open the sunroof" without receiving feedback, the local NLU can recognize the "verb + noun" sentence structure and provide a high-confidence local processing result, thus eliminating the need to retrieve the cloud processing result. However, in different embodiments, the voice stream can still be uploaded to the cloud simultaneously, or the ASR recognition result based on a weak network can still be uploaded.

[0058] It should be understood that when adjusting uplink and downlink data transmission volumes for weak network conditions, it is necessary to promptly return to normal uplink and downlink data transmission once network conditions are restored. For example, during the intervals when a user is transmitting voice data, the time between voice stream transmission and cloud processing result return can be calculated in the background, and the result can be used to determine whether it is necessary to return to the normal local-cloud dual-process state.

[0059] Furthermore, the present invention can also be implemented as a voice processing device. Figure 6 A schematic diagram of a voice processing device according to an embodiment of the present invention is shown. This device can be integrated into existing intelligent voice systems, such as as part of a home voice interaction system or an in-vehicle voice assistant. As shown, the voice processing device 600 may include a voice information acquisition unit 610, a local voice processing engine 620, a communication unit 630, an arbitration unit 640, and an operation unit 650. These units can be individual software functional modules or their corresponding functions can be implemented by specific hardware modules.

[0060] Specifically, the voice information obtaining unit 610 is configured to obtain voice information. In different scenarios, the voice information obtaining unit 610 can have different forms. In the most common application, the voice information obtaining unit 610 can be a voice collection unit, such as a microphone, which is built-in or external to the device. In a separate voice collection system, voice collection can be performed by a separate voice collection interface, and the voice information obtaining unit 610 can be a communication unit, such as a Bluetooth unit, which communicates with the voice collection interface at a short distance.

[0061] The local voice processing engine 620 is configured to perform local voice processing on the obtained voice information. The local voice processing engine 620 can include an ASR module and an NLU module, and can be implemented by a separate chip. The communication unit 630 is configured to input the obtained voice information to a cloud voice processing engine, and can be implemented as a WiFi unit. The arbitration unit 640 can be configured to determine the voice processing result to be used according to an offline-online arbitration rule. The arbitration unit 640 can implement the arbitration rule as shown in the following table, for example. Figure 4 The operation unit 650 is configured to perform corresponding operations based on the determined voice processing result. Here, the operation unit 650 can be a control unit of a device, such as a central control unit of a vehicle, configured to perform operations, such as opening a sunroof, according to the received processing result.

[0062] Here, although the voice processing procedures of the local voice processing engine and the cloud voice processing engine are still performed simultaneously via the local voice processing engine 620 and the communication unit 630, respectively.

[0063] In one embodiment, the arbitration unit 640 can be configured to, in a case where the cloud processing result is returned first, use the cloud processing result as the voice processing result to be used, and the local voice processing engine cancels the current processing procedure for the voice information.

[0064] In one embodiment, the arbitration unit 640 can be configured to, in a case where the local processing result is returned first, determine the voice processing result to be used according to the confidence of the local processing result.

[0065] In one embodiment, the arbitration unit 640 can be configured to, when the confidence of the local processing result is higher than a first threshold, use the local processing result as the voice processing result to be used.

[0066] In one embodiment, the arbitration unit 640 can be configured to: when the confidence of the local processing result is lower than a first threshold value and higher than a second threshold value, continue to wait for the cloud processing result for a first predetermined time; if the cloud processing result is returned within the first predetermined time, use the cloud processing result as the speech processing result to be determined; and if the cloud processing result is not returned within the first predetermined time, use the local processing result as the speech processing result to be determined.

[0067] In one embodiment, the arbitration unit 640 can be configured to: when the confidence of the local processing result is lower than a second threshold value, continue to wait for the cloud processing result for a second predetermined time; if the cloud processing result is returned within the second predetermined time, use the cloud processing result as the speech processing result to be determined; and if the cloud processing result is not returned within the second predetermined time, trigger a timeout process.

[0068] As mentioned above, the local processing engine 620 can include a local speech recognition (ASR) module and a local natural language processing (NLU) module. Under predetermined conditions, the communication unit 630 can upload the local speech recognition result generated by the local speech recognition module to the cloud speech processing engine as a substitute for the speech information; and / or under predetermined conditions, cause the cloud speech processing engine to issue a cloud processing result with a minimum amount of data, for example, by sending a request through the communication unit 630.

[0069] Similarly, the predetermined conditions can include: no cloud processing result is received within a third predetermined time; and / or the confidence of the local processing result is lower than a third threshold value.

[0070] It should be understood that "first", "second", "third" in the present application are only used to distinguish the same objects, and do not imply their front or back or important order.

[0071] Further, the present application can also be implemented as a speech processing system 700. Figure 7 A composition diagram of a speech processing system capable of implementing the present application is shown. As shown in the figure, the system 700 can include a plurality of speech processing devices 710 and a server 720. In other words, the server 720 can be regarded as a speech service platform for providing speech services for a large number of speech processing devices 710.

[0072] In the speech processing devices connected with the server 720, at least part of them can be configured to execute the speech processing method of the present application as described above in combination with the accompanying drawings. The server 720 includes a cloud speech processing engine, and is configured to issue a cloud speech processing result based on the obtained speech information.

[0073] The voice processing device can execute the voice processing method after being activated by a wake word. The voice processing method of the present invention is particularly suitable for devices that frequently operate in weak network conditions, such as portable devices. In one embodiment, the voice processing device can be, in particular, an in-vehicle voice assistant.

[0074] Furthermore, the same vehicle may include multiple in-vehicle voice assistants, each activated by a different wake word, and these multiple in-vehicle voice assistants connect to different servers after being activated to obtain cloud processing results.

[0075] Figure 8 A schematic diagram of a computing device that can be used to implement the above-described speech processing method according to an embodiment of the present invention is shown.

[0076] See Figure 8 The computing device 800 includes a memory 810 and a processor 820.

[0077] Processor 820 may be a multi-core processor or may contain multiple processors. In some embodiments, processor 820 may include a general-purpose main processor and one or more special-purpose coprocessors, such as a graphics processing unit (GPU), a digital signal processor (DSP), etc. In some embodiments, processor 820 may be implemented using custom circuitry, such as an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA).

[0078] The memory 810 can include various types of storage units, such as a system memory, a read-only memory (ROM), and a permanent storage device. Among them, the ROM can store static data or instructions required by the processor 820 or other modules of the computer. The permanent storage device can be a read-write storage device. The permanent storage device can be a non-volatile storage device that does not lose stored instructions and data even after the computer is powered off. In some embodiments, the permanent storage device uses a mass storage device (such as a magnetic or optical disk, a flash memory) as a permanent storage device. In some other embodiments, the permanent storage device can be a removable storage device (such as a floppy disk, an optical drive). The system memory can be a read-write storage device or a volatile read-write storage device, such as a dynamic random access memory. The system memory can store some or all of the instructions and data required by the processor during runtime. In addition, the memory 810 can include a combination of any computer readable storage media, including various types of semiconductor storage chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), magnetic disks and / or optical disks. In some embodiments, the memory 810 can include a read and / or write removable storage device, such as a compact disc (CD), a read-only digital versatile disc (such as DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (such as an SD card, a min SD card, a Micro-SD card, etc.), a magnetic floppy disk, etc. The computer readable storage medium does not include a carrier wave and a transient electronic signal transmitted through wireless or wired transmission.

[0079] The executable code stored on the memory 810 can cause the processor 820 to perform the voice processing method described above when the executable code is processed by the processor 820.

[0080] The voice processing method, device and system according to the present application have been described in detail above with reference to the accompanying drawings. The present application provides a set of online and offline voice processing result selection scheme. The scheme simultaneously performs local and cloud voice processing procedures, and makes real-time decisions according to the timeliness and confidence of the processing results, and can always provide accurate and timely voice processing results for users.

[0081] In addition, the method according to the present application can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing each step defined in the above method of the present application.

[0082] Alternatively, the present application can also be embodied as a non-transitory machine readable storage medium (or computer readable storage medium, or machine readable storage medium) on which executable code (or computer program, or computer instruction code) is stored, which when executed by a processor of an electronic device (or computing device, server, etc.) causes the processor to perform the various steps of the above-described method according to the present application.

[0083] Those skilled in the art will further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or combinations of both.

[0084] The flow diagrams and block diagrams in the drawings are representative of the architectural, functional, and operational aspects of possible implementations of systems and methods in accordance with the present application. In this regard, each block in the flow diagrams and block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0085] Embodiments of the present application have been described above, and the description is intended to be illustrative of the embodiments and not exhaustive, and is not limited to the embodiments disclosed. Numerous modifications and adaptations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The scope of the embodiments disclosed herein covers all technical equivalents that perform similar functions, have similar structures, or achieve similar results. The selection of the terms to be used in the description is intended to best explain the principles of the embodiments, the practical application, or the improvement over the technology in the art, or to enable other ordinary skilled persons in the art to understand the embodiments disclosed herein.

Claims

1. A method for processing voice, comprising: obtaining voice information; inputting the obtained voice information into a cloud voice processing engine and a local voice processing engine; determining a voice processing result to be used according to an off-line and on-line arbitration rule; and performing a corresponding operation based on the determined voice processing result; wherein the method further comprises: under a predetermined condition, causing the cloud voice processing engine to issue a cloud processing result of a minimum data volume version; and wherein the predetermined condition comprises: no cloud processing result is received within a third predetermined time; and a confidence level of the local processing result is lower than a third threshold value; wherein the third predetermined time is determined according to a regular waiting time when the network is normal. The inputting of the obtained voice information into the cloud voice processing engine and the local voice processing engine comprises:

2. The method of claim 1, wherein, sending the voice information to the cloud voice processing engine while sending the voice information to the local voice processing engine. The determining of the voice processing result to be used according to the off-line and on-line arbitration rule comprises:

3. The method of claim 1, wherein, in a case where the cloud processing result is returned first, using the cloud processing result as the voice processing result to be used. 4.The method of claim 3, further comprising: canceling a current processing flow of the local voice processing engine for the voice information. The determining of the voice processing result to be used according to the off-line and on-line arbitration rule comprises:

5. The method of claim 1, wherein, in a case where the local processing result is returned first, determining the voice processing result to be used according to a confidence level of the local processing result. The determining of the voice processing result to be used according to the confidence level of the local processing result in a case where the local processing result is returned first comprises:

6. The method of claim 5, wherein, when the confidence level of the local processing result is higher than a first threshold value, using the local processing result as the voice processing result to be used. The determining of the voice processing result to be used according to the confidence level of the local processing result in a case where the local processing result is returned first comprises:

7. The method of claim 6, wherein, when the confidence level of the local processing result is lower than the first threshold value and higher than a second threshold value, continuing to wait for the cloud processing result until a first predetermined time; if the cloud processing result is returned within the first predetermined time, using the cloud processing result as the voice processing result to be used; and if the cloud processing result is not returned within the first predetermined time, using the local processing result as the voice processing result to be used. The determining of the voice processing result to be used according to the confidence level of the local processing result in a case where the local processing result is returned first comprises:

8. The method of claim 7, wherein, when the confidence level of the local processing result is lower than the second threshold value, continuing to wait for the cloud processing result until a second predetermined time; if the cloud processing result is returned within the second predetermined time, using the cloud processing result as the voice processing result to be used; and if the cloud processing result is not returned within the second predetermined time, triggering a timeout processing. The local processing engine comprises a local voice recognition module and a local natural language processing module, and the method further comprises:

9. The method of claim 1, wherein, ​ Under a predetermined condition, the local speech recognition result generated by the local speech recognition module is uploaded to the cloud speech processing engine as a substitute for the speech information.

10. The method of claim 1, wherein, The speech information acquisition includes: Acquiring the speech information received after the speech assistant is woken up.

11. The method of claim 10, further comprising: Different speech assistants are woken up based on different wake-up words, and the acquired speech information is input into the cloud speech processing engine and the local speech processing engine, which includes: Inputting the acquired speech information into the same local speech processing engine and the cloud speech processing engine corresponding to the woken-up speech assistant.

12. A speech processing apparatus, comprising: a speech information acquisition unit configured to acquire speech information; a local speech processing engine configured to perform local speech processing on the acquired speech information; a communication unit configured to input the acquired speech information into a cloud speech processing engine; an arbitration unit configured to determine a speech processing result to be used according to an online-offline arbitration rule; and an operation unit configured to perform a corresponding operation based on the determined speech processing result; wherein the communication unit is further configured to: under a predetermined condition, causing the cloud speech processing engine to issue a cloud processing result with a minimum data volume; wherein the predetermined condition includes: no cloud processing result is received within a third predetermined time; and the confidence of the local processing result is lower than a third threshold value; wherein the third predetermined time is determined according to a regular waiting time when the network is normal.

13. The apparatus of claim 12, wherein, The speech processing procedures of the local speech processing engine and the cloud speech processing engine are performed simultaneously.

14. The apparatus of claim 12, wherein, The arbitration unit is configured to: in the case that the cloud processing result is returned first, use the cloud processing result as the speech processing result to be used, and the local speech processing engine cancels the current processing procedure for the speech information.

15. The apparatus of claim 12, wherein the arbitration unit is configured to: in the case that the local processing result is returned first, determine the speech processing result to be used according to the confidence of the local processing result.

16. The apparatus of claim 15, wherein, The arbitration unit is configured to: when the confidence of the local processing result is higher than a first threshold value, use the local processing result as the speech processing result to be used.

17. The apparatus of claim 16, wherein, The arbitration unit is configured to: when the confidence of the local processing result is lower than the first threshold value and higher than a second threshold value, continue to wait for the cloud processing result until a first predetermined time; if the cloud processing result is returned within the first predetermined time, use the cloud processing result as the speech processing result to be used; and if the cloud processing result is not returned within the first predetermined time, use the local processing result as the speech processing result to be used.

18. The apparatus of claim 17, wherein, The arbitration unit is configured to: when the confidence of the local processing result is lower than the second threshold value, continue to wait for the cloud processing result until a second predetermined time; if the cloud processing result is returned within the second predetermined time, use the cloud processing result as the speech processing result to be used; and if the cloud processing result is not returned within the second predetermined time, trigger a timeout processing.

19. The apparatus of claim 12, wherein, The local processing engine includes a local speech recognition module and a local natural language processing module, And, under a predetermined condition, the communication unit uploads, as a substitute for the voice information, a local speech recognition result generated by the local speech recognition module to the cloud speech processing engine. 20.A voice processing system, comprising: a voice processing device configured to perform the voice processing method of any one of claims 1-11; and a server including the cloud speech processing engine and configured to issue a cloud speech processing result based on the acquired voice information. The voice processing device performs the voice processing method after being woken up by a wake-up word.

21. The system of claim 20, wherein, The voice processing device is a vehicle-mounted voice assistant.

22. The system of claim 20, wherein, Multiple vehicle-mounted voice assistants on the same vehicle are included, each of which is woken up by a different wake-up word, and the multiple vehicle-mounted voice assistants connect different servers to acquire cloud processing results after being woken up.

23. The system of claim 20, wherein, 24.A computing device, comprising: a processor; and a memory having stored thereon executable code that, when executed by the processor, causes the processor to perform the method of any one of claims 1-11. 25.A non-transitory machine-readable storage medium having stored thereon executable code that, when executed by a processor of an electronic device, causes the processor to perform the method of any one of claims 1-11. ​

Images