Semantic recognition method and device for audio stream data, storage medium and electronic device

Abstract

The invention discloses a semantic recognition method and device for audio stream data, a storage medium and an electronic device, and relates to the technical field of smart home, and the method comprises the steps: obtaining a first recognition result obtained when automatic speech recognition ASR is carried out on the audio stream data from terminal equipment; when it is monitored that an end identifier of a voice activity detection VAD threshold value appears in the first recognition result, obtaining a to-be-processed audio corresponding to the end identifier of the VAD threshold value from the audio stream data, determining a first intention corpus corresponding to the to-be-processed audio, and caching an intention recognition result of the first intention corpus; after the ASR of the audio stream data is determined to be completed, obtaining a second identification result of the audio stream data; the target intention corresponding to the audio stream data is determined from the cached intention recognition result according to the verification result of the first intention corpus and the second intention corpus, and by adopting the technical scheme, the technical problem that the time consumed for recognizing the audio data is too long, and consequently the time consumed for voice interaction is large is solved.

Description

Technical Field

[0001] The present application relates to the field of smart home technology, and more specifically, to a method, device, storage medium and electronic device for semantic recognition of audio stream data. Background Art

[0002] At present, speech recognition technology is widely used in fields such as smart home, and plays an important role in voice interaction and other aspects. Generally speaking, during speech recognition, the audio transmitted on the terminal will enter the semantic analysis stage of natural language processing (NLP) after being recognized by automatic speech recognition (Automatic Speech Recognition, ASR), and then determine the subsequent processing steps based on the analysis conclusion. This conventional voice interaction process needs to go through ASR recognition, NLP analysis and subsequent business processing until broadcasting. Since the user's voice needs to be fully processed by ASR before NLP processing, it takes too long to recognize the audio data, resulting in a long voice interaction time, which brings a poor interaction experience to users.

[0003] Therefore, in the related art, there is a technical problem that it takes too long to recognize audio data, resulting in a long time consumption of voice interaction.

[0004] Regarding the technical problem in related technologies that it takes too long to recognize audio data, resulting in time-consuming voice interaction, no effective solution has yet been proposed. Summary of the invention

[0005] The embodiments of the present application provide a method, device, storage medium and electronic device for semantic recognition of audio stream data, so as to at least solve the technical problem in the related art that it takes too long to recognize audio data, resulting in time-consuming voice interaction.

[0006] According to an embodiment of the present application, a semantic recognition method for audio stream data is provided, including: obtaining a first recognition result obtained when automatic speech recognition (ASR) is performed on audio stream data from a terminal device; when an end mark of a voice activity detection (VAD) threshold appears in the first recognition result, obtaining a to-be-processed audio corresponding to the end mark of the VAD threshold from the audio stream data, determining a first intent corpus corresponding to the to-be-processed audio, and caching an intent recognition result of the first intent corpus; after determining that the ASR of the audio stream data is completed, obtaining a second recognition result of the audio stream data, wherein the second recognition result includes at least a second intent corpus corresponding to the audio stream data; and determining a target intent corresponding to the audio stream data from the cached intent recognition results according to the verification results of the first intent corpus and the second intent corpus, wherein the target intent is used to represent the semantics corresponding to the audio stream data.

[0007] In an exemplary embodiment, a first recognition result obtained when automatic speech recognition ASR is performed on audio stream data from a terminal device is obtained, including: performing speech recognition on the audio stream data to obtain speech data and non-speech data; for the speech data, intercepting the speech data based on a start identifier of the VAD threshold and an end identifier of the VAD threshold to obtain intercepted speech; and performing text recognition on the intercepted speech to obtain the first recognition result.

[0008] In an exemplary embodiment, the target intent corresponding to the audio stream data is determined from the cached intent recognition results based on the verification results of the first intent corpus and the second intent corpus, including: when it is determined that the verification result indicates that the first intent corpus and the second intent corpus are consistent, obtaining the preprocessing progress of natural language processing on the first intent corpus; when it is determined that the preprocessing progress is completed, determining the first intent obtained by natural language processing on the first intent corpus as the target intent; when it is determined that the verification result indicates that the first intent corpus and the second intent corpus are inconsistent, obtaining the second intent obtained by natural language processing on the second intent corpus from the cached intent recognition results, and determining the second intent as the target intent.

[0009] In an exemplary embodiment, after obtaining the preprocessing progress of natural language processing on the first intent corpus, the method further includes: when it is determined that the preprocessing progress is not completed, continuing to monitor the preprocessing progress of natural language processing on the first intent corpus until the preprocessing progress is completed; obtaining a first processing time for natural language processing on the first intent corpus, and when it is determined that the first processing time is greater than a preset time, obtaining a third intent obtained after natural language processing on the second intent corpus, and determining the third intent as the target intent.

[0010] In an exemplary embodiment, after obtaining the preprocessing progress of natural language processing on the first intent corpus, the method further includes: when it is determined that the preprocessing progress is not completed, continuing to monitor the preprocessing progress of natural language processing on the first intent corpus, and obtaining a second processing time for natural language processing on the first intent corpus; when it is determined that the second processing time is less than a preset time, determining the first intent as the target intent.

[0011] In an exemplary embodiment, after determining the first intent obtained by performing natural language processing on the first intent corpus as the target intent, the method also includes: determining the device number of the terminal device and obtaining a device list corresponding to the device number; determining different terminal devices corresponding to all terminal device numbers in the device list, and selecting a target terminal device with a target device type from the different terminal devices, wherein the target terminal device represents a device for sending a response message generated based on the target intent to a target object.

[0012] In an exemplary embodiment, after determining the first intent obtained by performing natural language processing on the first intent corpus as the target intent, the method further includes: respectively obtaining the audio format of the audio to be processed, the audio integrity of the audio to be processed, the signal-to-noise ratio of the audio to be processed, and the frequency spectrum of the audio to be processed; and performing a rejection operation on the audio to be processed based on the audio format of the audio to be processed, the audio integrity of the audio to be processed, the signal-to-noise ratio of the audio to be processed, and the frequency spectrum of the audio to be processed.

[0013] According to another aspect of an embodiment of the present application, a semantic recognition device for audio stream data is also provided, including: an obtaining module, used to obtain a first recognition result obtained when automatic speech recognition ASR is performed on audio stream data from a terminal device; a first determination module, used to obtain, when monitoring the appearance of an end mark of a voice activity detection VAD threshold in the first recognition result, an audio to be processed corresponding to the end mark of the VAD threshold from the audio stream data, determine a first intent corpus corresponding to the audio to be processed, and cache the intent recognition result of the first intent corpus; an obtaining module, used to obtain a second recognition result of the audio stream data after determining that the ASR of the audio stream data is completed, wherein the second recognition result includes at least a second intent corpus corresponding to the audio stream data; a second determination module, used to determine a target intent corresponding to the audio stream data from the cached intent recognition results according to the verification results of the first intent corpus and the second intent corpus, wherein the target intent is used to represent the semantics corresponding to the audio stream data.

[0014] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned method for semantic recognition of audio stream data when running.

[0015] According to another aspect of an embodiment of the present application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the above-mentioned method for semantic recognition of audio stream data through the computer program.

[0016] In an embodiment of the present application, a first recognition result obtained when automatic speech recognition ASR is performed on audio stream data from a terminal device is obtained; when the end mark of the voice activity detection VAD threshold appears in the first recognition result, the audio to be processed corresponding to the end mark of the VAD threshold is obtained from the audio stream data, the first intention corpus corresponding to the audio to be processed is determined, and the intention recognition result of the first intention corpus is cached; after determining that the ASR of the audio stream data is completed, the second recognition result of the audio stream data is obtained, wherein the second recognition result at least includes the second intention corpus corresponding to the audio stream data; according to the verification results of the first intention corpus and the second intention corpus, the target intention corresponding to the audio stream data is determined from the cached intention recognition results, and the target intention is used to represent the semantics corresponding to the audio stream data. On the one hand, the present application caches the intention recognition results of some of the intention corpora that are first recognized (for example, cached to the intention cache library), realizes a preprocessing mechanism, can reduce repeated processing, and improve the efficiency of voice interaction. On the other hand, the present application improves the accuracy and response speed of continuous speech recognition through automatic speech recognition. ASR can be used to process the audio stream in real time, quickly give preliminary recognition results, and then provide more comprehensive recognition results after the audio stream ends. By comparing the intent corpus of the two, misrecognition can be effectively avoided and the accuracy of the recognition results can be ensured. Therefore, the above technical solution solves the technical problem that the time required to recognize audio data is too long, resulting in a long voice interaction time, thereby improving the recognition speed, reducing the time of voice interaction, and improving the efficiency of voice interaction. In practical applications, this method also significantly improves the user experience, especially in scenarios such as intelligent voice assistants and voice recognition systems, which can quickly and accurately respond to user needs and has important application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 It is a hardware environment diagram of a semantic recognition method for audio stream data according to an embodiment of the present application;

[0020] Figure 2 is a flowchart of a method for semantic recognition of audio stream data according to an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of a semantic recognition process of audio stream data according to an embodiment of the present application;

[0022] Figure 4 is a flowchart of a method for semantic recognition of audio stream data according to an embodiment of the present application;

[0023] Figure 5 It is a structural block diagram of a semantic recognition device for audio stream data according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] According to one aspect of an embodiment of the present application, a method for semantic recognition of audio stream data is provided. The method for semantic recognition of audio stream data is widely used in smart home, smart home, smart home device ecology, smart residential (Intelligence House) ecology and other whole-house intelligent digital control application scenarios. Optionally, in this embodiment, the above-mentioned method for semantic recognition of audio stream data can be applied to Figure 1 In the hardware environment composed of the terminal device 102 and the server 104 shown in FIG. Figure 1As shown, the server 104 is connected to the terminal device 102 via a network, and can be used to provide services (such as application services, etc.) for the terminal or a client installed on the terminal. A database can be set on the server or independently of the server to provide data storage services for the server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data computing services for the server 104.

[0027] The network may include but is not limited to at least one of the following: wired network, wireless network. The wired network may include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the wireless network may include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal device 102 may be but is not limited to a PC, a mobile phone, a tablet computer, a smart air conditioner, a smart range hood, a smart refrigerator, a smart oven, a smart stove, a smart washing machine, a smart water heater, a smart washing device, a smart dishwasher, a smart projection device, a smart TV, a smart clothes drying rack, a smart curtain, a smart audio and video, a smart socket, a smart speaker, a smart fresh air device, a smart kitchen and bathroom device, a smart bathroom device, a smart sweeping robot, a smart window cleaning robot, a smart mopping robot, a smart air purification device, a smart steamer, a smart microwave oven, a smart kitchen treasure, a smart purifier, a smart water dispenser, a smart door lock, etc.

[0028] In this embodiment, a semantic recognition method for audio stream data is provided, which is applied to the above terminal device. Figure 2 : is a flowchart of a method for semantic recognition of audio stream data according to an embodiment of the present application, the process comprising the following steps:

[0029] Step S202, obtaining a first recognition result obtained when automatic speech recognition ASR is performed on the audio stream data from the terminal device;

[0030] Step S204, when it is monitored that the first recognition result has an end mark of the voice activity detection VAD threshold, obtaining the to-be-processed audio corresponding to the end mark of the VAD threshold from the audio stream data, determining the first intent corpus corresponding to the to-be-processed audio, and caching the intent recognition result of the first intent corpus;

[0031] Step S206, after determining that the ASR of the audio stream data is completed, obtaining a second recognition result of the audio stream data, wherein the second recognition result at least includes a second intent corpus corresponding to the audio stream data;

[0032] Optionally, the first recognition result can also be obtained through streaming recognition. Streaming recognition is when the audio stream data (voice signal) is continuously transmitted into the system, and the system outputs the recognition result in real time. Streaming recognition has good real-time performance and can output the recognition result in real time, but its recognition performance is usually slightly lower than that of automatic speech recognition.

[0033] Optionally, for the second recognition result, the audio stream data may be recorded first, and then the speech recognition result may be output.

[0034] Furthermore, the present application can perform recognition by combining streaming recognition and automatic speech recognition to jointly improve the real-time and accuracy of the recognition results.

[0035] Step S208: determining a target intent corresponding to the audio stream data from the cached intent recognition results according to the verification results of the first intent corpus and the second intent corpus, wherein the target intent is used to represent the semantics corresponding to the audio stream data.

[0036] Through the above steps, the first recognition result obtained when the automatic speech recognition ASR is performed on the audio stream data from the terminal device is obtained; when the end mark of the voice activity detection VAD threshold appears in the first recognition result, the audio to be processed corresponding to the end mark of the VAD threshold is obtained from the audio stream data, the first intention corpus corresponding to the audio to be processed is determined, and the intention recognition result of the first intention corpus is cached; after determining that the ASR of the audio stream data is completed, the second recognition result of the audio stream data is obtained, wherein the second recognition result at least includes the second intention corpus corresponding to the audio stream data; according to the verification results of the first intention corpus and the second intention corpus, the target intention corresponding to the audio stream data is determined from the cached intention recognition results. On the one hand, the present application caches the intention recognition results of the first recognized part of the intention corpus (for example, cached to the intention cache library), realizes the preprocessing mechanism, can reduce repeated processing, and improve the efficiency of voice interaction. On the other hand, the present application improves the accuracy and response speed of continuous speech recognition through automatic speech recognition. The audio stream can be processed in real time by ASR first, and a preliminary recognition result can be quickly given, and then a more comprehensive recognition result can be provided after the audio stream ends. By comparing the intent corpora of the two, it is possible to effectively avoid misrecognition and ensure the accuracy of the recognition results. Therefore, the above technical solution solves the technical problem that it takes too long to recognize audio data, resulting in a long voice interaction time, thereby improving the recognition speed, reducing the time consumption of voice interaction, and improving the efficiency of voice interaction. In practical applications, this method also significantly improves the user experience, especially in scenarios such as intelligent voice assistants and voice recognition systems, which can quickly and accurately respond to user needs and has important application value.

[0037] In the general audio recognition process, after the ASR processing is completed, it is necessary to wait for other operations such as business processing, voiceprint recognition, error correction, and then call NLP for intent analysis. These steps take up a lot of time. Considering that NLP is only used to process audio and corpus during actual processing, and is not strongly associated with other processing links such as business configuration, the NLP processing order can be appropriately adjusted to save time waiting for the NLP processing to complete. Therefore, this application starts NLP processing after the preliminary recognition result, instead of waiting for ASR to be completely completed before NLP processing. At the same time, the processing time of other businesses can be carried out in parallel with NLP. In this way, NLP is called directly after initially obtaining relatively accurate results, and NLP and subsequent business processing operations can be processed in parallel, and the cache mechanism is used to store the results of the advance processing, which is convenient for subsequent direct acquisition or proofreading, thereby constructing a complete process from the end of VAD to NLP processing, and the NLP advance processing is used to achieve the technical purpose of efficiently and accurately processing the data after speech recognition, and providing users with the required services, thereby compressing the overall required time.

[0038] Optionally, in one embodiment, the processing process of the above other services is described:

[0039] For example, a device list can be pulled from a database or device registration service based on the device number of the requested device. It is also possible to use a pre-trained language model or a custom intent recognition algorithm to perform intent analysis on the recognized text to determine the user's intent. Alternatively, audio data rejection judgment can be performed, specifically performing audio quality checks and background noise analysis on the recognized audio data to determine whether rejection is required.

[0040] Alternatively, the results of NLP processing (including intent recognition results, audio rejection judgments, etc.) can be cached in memory. This can be achieved by using an in-memory database, a cache system, or an in-memory data structure. Once the NLP processing is completed, the processing results are fed back to the requesting device or related system, and then a message is sent over the network, the user interface is updated, or other system processes are triggered.

[0041] In addition, error handling mechanisms can be implemented throughout the process to deal with any possible abnormal situations. Logs are recorded to facilitate problem tracking and system monitoring. And, considering that NLP processing may take a long time, asynchronous processing or multi-threading / multi-processing can be implemented to improve system response speed and thus optimize performance.

[0042] Of course, we must also ensure that user privacy is protected during the processing process and improve the security of data processing. And before actual deployment, we must conduct sufficient testing including unit testing, integration testing, and performance testing to ensure the stability and accuracy of the system.

[0043] In an exemplary embodiment, a process for implementing the acquisition of a first recognition result obtained when automatic speech recognition ASR is performed on audio stream data from a terminal device is proposed, including: performing speech recognition on the audio stream data to obtain speech data and non-speech data; for the speech data, intercepting the speech data based on the start identifier of the VAD threshold and the end identifier of the VAD threshold to obtain intercepted speech; performing text recognition on the intercepted speech to obtain the first recognition result. The present application intercepts speech by using the start identifier of the VAD threshold and the end identifier of the VAD threshold, so as to accurately identify effective speech and improve recognition accuracy and efficiency.

[0044] It should be noted that the end mark VADend of VAD (Voice Activity Detection) indicates the moment when the voice activity detection process ends, that is, the moment when the voice signal ends. The effective voice part in the voice signal can be detected by VAD according to the start mark of the VAD threshold and the end mark of the VAD threshold, thereby improving the accuracy and efficiency of recognition.

[0045] Specifically, for example, the VAD module is used to analyze the input speech signal to identify the speech part and the non-speech part (such as silence, background noise, etc.). The start and end of the speech are determined according to the start time (VAD start) and end time (VAD end) of the detected speech signal, so that only the valid speech part is processed and converted into text information.

[0046] In an exemplary embodiment, the technical solution for determining the target intent corresponding to the audio stream data from the cached intent recognition results according to the verification results of the first intent corpus and the second intent corpus is described by the following steps: when it is determined that the verification result indicates that the first intent corpus and the second intent corpus are consistent, the preprocessing progress of the first intent corpus for natural language processing is obtained; when it is determined that the preprocessing progress is completed, the first intent obtained by natural language processing of the first intent corpus is determined as the target intent; when it is determined that the verification result indicates that the first intent corpus and the second intent corpus are inconsistent, the second intent obtained by natural language processing of the second intent corpus is obtained from the cached intent recognition results, and the second intent is determined as the target intent. Among them, the process of obtaining the first intent by natural language processing of the first intent corpus can be understood as the intent preprocessing process.

[0047] In an exemplary embodiment, after obtaining the preprocessing progress of natural language processing of the first intent corpus, further, when it is determined that the preprocessing progress is not completed, continue to monitor the preprocessing progress of natural language processing of the first intent corpus until the preprocessing progress is completed; obtain the first processing time of natural language processing of the first intent corpus, and when it is determined that the first processing time is greater than a preset time, obtain the third intent obtained after natural language processing of the second intent corpus, and determine the third intent as the target intent. Optionally, when the first processing time is less than a preset time, determine the first intent as the target intent.

[0048] According to the above embodiment, if the waiting time for the preprocessing of the first intent language has expired, the second intent language is used for parsing. Otherwise, the preprocessing result of the first intent can be directly obtained from the intent cache.

[0049] In an exemplary embodiment, after obtaining the preprocessing progress of natural language processing on the first intent corpus, further, when it is determined that the preprocessing progress is not completed, the preprocessing progress of natural language processing on the first intent corpus can continue to be monitored, and a second processing time for natural language processing on the first intent corpus can be obtained; when it is determined that the second processing time is less than a preset time, the first intent is determined as the target intent.

[0050] Optionally, a third intent is obtained after natural language processing is performed on the second intent corpus, and the third intent is determined as the target intent.

[0051] In an exemplary embodiment, after determining the first intent obtained by performing natural language processing on the first intent corpus as the target intent, the device number of the terminal device can be determined, and a device list corresponding to the device number can be obtained; different terminal devices corresponding to all terminal device numbers in the device list can be determined, and a target terminal device with a target device type can be selected from the different terminal devices, wherein the target terminal device is used to send a response message generated based on the target intent to the target object.

[0052] It should be noted that the target device type is, for example, an audio device type, a visual device type, or other device.

[0053] In an exemplary embodiment, after the first intent obtained by performing natural language processing on the first intent corpus is determined as the target intent, the audio format of the audio to be processed, the audio integrity of the audio to be processed, the signal-to-noise ratio of the audio to be processed, and the frequency spectrum of the audio to be processed can be obtained respectively; and the audio to be processed can be rejected based on the audio format of the audio to be processed, the audio integrity of the audio to be processed, the signal-to-noise ratio of the audio to be processed, and the frequency spectrum of the audio to be processed.

[0054] It should be noted that in audio data processing, rejection means that the system automatically determines that some audio data does not meet the requirements or is of poor quality, so no further processing or analysis is performed. The following content describes the rejection scenario and the corresponding analysis method.

[0055] For audio quality inspection, the analysis process specifically includes:

[0056] Volume detection: If the audio volume is too low, the voice information may not be captured clearly, so it needs to be rejected.

[0057] Signal-to-noise ratio analysis: If the volume of background noise is similar to or higher than the volume of the speech signal, the signal-to-noise ratio is too low, which may lead to inaccurate speech recognition and need to be rejected.

[0058] Audio integrity check. If the audio file is corrupted or incomplete, it may need to be rejected.

[0059] Sampling rate and bit depth check: If the sampling rate or bit depth of the audio does not meet the system requirements, it may need to be rejected.

[0060] The analysis process for background noise includes:

[0061] Spectrum analysis: By analyzing the spectrum of the audio, non-speech components can be identified, such as continuous background noise, music, etc. If there are too many of these components, they may need to be rejected.

[0062] Sound event detection. Use sound event detection algorithms to identify and classify events in audio, such as doorbells, phone rings, etc. If these events are not relevant to the target speech recognition task, they may need to be rejected.

[0063] The analysis process for voice activity detection is:

[0064] Non-speech segment detection: The non-speech segments in the audio are detected by the VAD algorithm, such as long periods of silence or non-speech sounds, which need to be rejected.

[0065] For speakers, consistency checks are required: if the audio contains multiple speakers and the system only needs to recognize the voice of one speaker, it may be necessary to reject the parts containing the voices of other speakers.

[0066] It is necessary to identify whether the interactive language is not the target language. If the system only recognizes a specific language and the audio contains other languages, it may need to be rejected.

[0067] The analysis process for content-related rejection includes, for example, sensitive content detection: If the audio contains sensitive or inappropriate content, it may need to be rejected to comply with laws, regulations or platform policies.

[0068] You can also analyze the degree of processing capacity limitations: if the audio file is too large and exceeds the system's processing capacity, it may need to be rejected.

[0069] In order to better understand the process of the semantic recognition method of the above-mentioned audio stream data, the implementation method flow of the semantic recognition of the above-mentioned audio stream data is described below in combination with an optional embodiment, but it is not used to limit the technical solution of the embodiment of the present application.

[0070] In this embodiment, a method for semantic recognition of audio stream data is provided. Figure 3 is a schematic diagram of the semantic recognition process of audio stream data according to an embodiment of the present application, combined with the following steps and Figure 3 The mechanism of LP internal processing is shown to describe this process:

[0071] Step 1: Listen to the VAD end to call the intermediate streaming NLP.

[0072] For this step, during the voice interaction process, the client (corresponding to the above-mentioned mobile terminal) will continuously send the user's audio stream to the cloud, which performs streaming recognition. If VAD end is determined, NLP will be called for advance processing. At this time, the cloud will pass the necessary messages such as the sessionId, recognition text, request device, and recognized audio data of the session to NLP.

[0073] It is understandable that when VAD detects VAD end and determines that the voice activity ends, it will trigger an event or callback function to notify the cloud system to proceed to the next step. For example, after the VAD end signal is triggered, the NLP module is called for processing. The necessary information (session ID of the session, recognized text, requesting device, recognized audio data, etc.) is passed to the NLP module.

[0074] Step 2: The NLP module performs intent analysis.

[0075] The NLP module can call instructions for intermediate results, pull the device list according to the device number of the requested device, perform intent analysis on the recognized text, and determine whether the recognized audio data is rejected. The processing results will be temporarily stored in the memory.

[0076] Step 3: NLP result verification and return.

[0077] When the ASR recognition is completed as a whole (including subsequent business configuration query, voiceprint recognition, cloud configuration error correction, etc.), the NLP module will be called again for secondary processing. At this time, the NLP module compares the first recognized intent corpus (i.e., the first intent corpus) and the ASR recognized intent corpus (i.e., the second intent corpus). If they are consistent, it determines whether the preprocessing process based on the first recognized intent corpus is completed. If not completed, it will wait (for example, up to 400ms), otherwise it will be parsed according to the new intent recognized by ASR.

[0078] ASR technology is used in many fields, including voice assistants such as Siri, voice transcription services, voice-controlled devices, etc. The working principle of ASR system is as follows:

[0079] First, the sound signal is collected through a microphone, and then the collected sound signal is preprocessed, including noise reduction, echo removal, etc., to improve the accuracy of recognition. Then, features are extracted from the sound signal, which can represent the key information of the speech signal. And the extracted features are matched with known speech patterns using an acoustic model. The language model is used to help the system understand the grammatical structure of words and sentences and improve the contextual accuracy of recognition. Finally, the output of the acoustic model and the language model are combined to generate the most likely text representation. Some post-processing such as grammar correction and synonym replacement can be performed later to improve the quality of the final output.

[0080] like Figure 3 As shown in the figure, the time for calling the NLP module for the second time depends on the preprocessing progress of the NLP module on the first recognized intent corpus. For example, when the NLP module processes the intent, it takes 170ms. The first time the NLP preprocessing is called after ASR recognition, it takes a total of 170ms in parallel. Then, when calling NLP for the second time, you only need to get the intent result from the processing, and the time taken is generally no more than 10ms.

[0081] By adding the step of calling NLP preprocessing for the first time, approximately (1-100ms)+10+1+6+6+(20-50)=44-173ms can be saved without considering network jitter.

[0082] like Figure 4 As shown, the specific steps are as follows:

[0083] 1. NLP intent analysis.

[0084] 2. Determine whether the corpus is the same as the semi-stream processing corpus. If so, execute 3. Otherwise, execute 4.

[0085] 3. Read the parsing results from preprocessing.

[0086] 4. Re-parse the intent.

[0087] 5. Check whether the analysis is complete. If yes, execute 7; otherwise, execute 6.

[0088] 6. Wait for the intermediate streaming processing to complete.

[0089] 7. Get the results from preprocessing.

[0090] 8. Return the intent result.

[0091] Through the above steps, the parallel execution mindset can be used to transform the serial part into a parallel function, and the NLP intent parsing function that does not depend on other business processing can be processed in parallel with other business points in advance to improve the response efficiency of the interface. At the same time, compression is also achieved in the overall link delay, thereby bringing users a better experience.

[0092] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0093] Figure 5 is a structural block diagram of a semantic recognition device for audio stream data according to an embodiment of the present application; Figure 5 As shown, including:

[0094] An obtaining module 52 is used to obtain a first recognition result obtained when automatic speech recognition ASR is performed on the audio stream data from the terminal device;

[0095] A first determination module 54 is configured to, when an end mark of a voice activity detection (VAD) threshold appears in the first recognition result, obtain an audio to be processed corresponding to the end mark of the VAD threshold from the audio stream data, determine a first intent corpus corresponding to the audio to be processed, and cache an intent recognition result of the first intent corpus;

[0096] An acquisition module 56 is configured to acquire a second recognition result of the audio stream data after determining that the ASR of the audio stream data is completed, wherein the second recognition result at least includes a second intent corpus corresponding to the audio stream data;

[0097] The second determination module 58 is used to determine the target intent corresponding to the audio stream data from the cached intent recognition results according to the verification results of the first intent corpus and the second intent corpus, and the target intent is used to represent the semantics corresponding to the audio stream data.

[0098] Through the above-mentioned device, the first recognition result obtained when the automatic speech recognition ASR is performed on the audio stream data from the terminal device is obtained; when the end mark of the voice activity detection VAD threshold appears in the first recognition result, the audio to be processed corresponding to the end mark of the VAD threshold is obtained from the audio stream data, the first intention corpus corresponding to the audio to be processed is determined, and the intention recognition result of the first intention corpus is cached; after determining that the ASR of the audio stream data is completed, the second recognition result of the audio stream data is obtained, wherein the second recognition result at least includes the second intention corpus corresponding to the audio stream data; according to the verification results of the first intention corpus and the second intention corpus, the target intention corresponding to the audio stream data is determined from the cached intention recognition results. On the one hand, the present application caches the intention recognition results of the first recognized part of the intention corpus (for example, caches it to the intention cache library), realizes the preprocessing mechanism, can reduce repeated processing, and improve the efficiency of voice interaction. On the other hand, the present application improves the accuracy and response speed of continuous speech recognition through automatic speech recognition. The audio stream can be processed in real time by ASR first, and a preliminary recognition result can be quickly given, and then a more comprehensive recognition result can be provided after the audio stream ends. By comparing the intent corpora of the two, it is possible to effectively avoid misrecognition and ensure the accuracy of the recognition results. Therefore, the above technical solution solves the technical problem that it takes too long to recognize audio data, resulting in a long voice interaction time, thereby improving the recognition speed, reducing the time consumption of voice interaction, and improving the efficiency of voice interaction. In practical applications, this method also significantly improves the user experience, especially in scenarios such as intelligent voice assistants and voice recognition systems, which can quickly and accurately respond to user needs and has important application value.

[0099] In an exemplary embodiment, the obtaining module is also used to: perform speech recognition on the audio stream data to obtain speech data and non-speech data; for the speech data, intercept the speech data based on the start identifier of the VAD threshold and the end identifier of the VAD threshold to obtain intercepted speech; perform text recognition on the intercepted speech to obtain the first recognition result.

[0100] In an exemplary embodiment, the second determination module is also used to illustrate the technical solution of determining the target intent corresponding to the audio stream data from the cached intent recognition results based on the verification results of the first intent corpus and the second intent corpus through the following steps: when it is determined that the verification result is used to indicate that the first intent corpus and the second intent corpus are consistent, obtaining the preprocessing progress of natural language processing on the first intent corpus; when it is determined that the preprocessing progress is completed, determining the first intent obtained by natural language processing on the first intent corpus as the target intent; when it is determined that the verification result is used to indicate that the first intent corpus and the second intent corpus are inconsistent, obtaining the second intent obtained after natural language processing on the second intent corpus from the cached intent recognition results, and determining the second intent as the target intent.

[0101] In an exemplary embodiment, the second determination module is also used to, after obtaining the preprocessing progress of natural language processing on the first intent corpus, further, when it is determined that the preprocessing progress is not completed, continue to monitor the preprocessing progress of natural language processing on the first intent corpus until the preprocessing progress is completed; obtain the first processing time for natural language processing on the first intent corpus, and when it is determined that the first processing time is greater than the preset time, obtain the third intent obtained after natural language processing on the second intent corpus, and determine the third intent as the target intent.

[0102] In an exemplary embodiment, the second determination module is also used to, after obtaining the preprocessing progress of natural language processing on the first intent corpus, further, when it is determined that the preprocessing progress is not completed, continue to monitor the preprocessing progress of natural language processing on the first intent corpus and obtain a second processing time for natural language processing on the first intent corpus; when it is determined that the second processing time is less than a preset time, determine the first intent as the target intent.

[0103] In an exemplary embodiment, the second determination module is also used to determine the device number of the terminal device and obtain a device list corresponding to the device number after determining the first intent obtained by natural language processing of the first intent corpus as the target intent; determine different terminal devices corresponding to all terminal device numbers in the device list, and select a target terminal device with a target device type from the different terminal devices, wherein the target terminal device is used to send a response message generated based on the target intent to the target object.

[0104] In an exemplary embodiment, the second determination module is also used to obtain the audio format of the audio to be processed, the audio integrity of the audio to be processed, the signal-to-noise ratio of the audio to be processed, and the frequency spectrum of the audio to be processed, after determining the first intent obtained by natural language processing of the first intent corpus as the target intent; and perform a rejection operation on the audio to be processed based on the audio format of the audio to be processed, the audio integrity of the audio to be processed, the signal-to-noise ratio of the audio to be processed, and the frequency spectrum of the audio to be processed.

[0105] An embodiment of the present application further provides a storage medium, which includes a stored program, wherein the program executes any of the above methods when it is run.

[0106] Optionally, in this embodiment, the storage medium may be configured to store program codes for executing the following steps:

[0107] S1, obtaining a first recognition result obtained when automatic speech recognition ASR is performed on audio stream data from a terminal device;

[0108] S2, when it is detected that the first recognition result has an end mark of the voice activity detection (VAD) threshold, obtaining the to-be-processed audio corresponding to the end mark of the VAD threshold from the audio stream data, determining the first intent corpus corresponding to the to-be-processed audio, and caching the intent recognition result of the first intent corpus;

[0109] S3, after determining that the ASR of the audio stream data is completed, obtaining a second recognition result of the audio stream data, wherein the second recognition result at least includes a second intent corpus corresponding to the audio stream data;

[0110] S4: Determine a target intent corresponding to the audio stream data from cached intent recognition results according to verification results of the first intent corpus and the second intent corpus, where the target intent is used to represent semantics corresponding to the audio stream data.

[0111] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0112] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0113] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0114] S1, obtaining a first recognition result obtained when automatic speech recognition ASR is performed on audio stream data from a terminal device;

[0115] S2, when it is detected that the first recognition result has an end mark of the voice activity detection (VAD) threshold, obtaining the to-be-processed audio corresponding to the end mark of the VAD threshold from the audio stream data, determining the first intent corpus corresponding to the to-be-processed audio, and caching the intent recognition result of the first intent corpus;

[0116] S3, after determining that the ASR of the audio stream data is completed, obtaining a second recognition result of the audio stream data, wherein the second recognition result at least includes a second intent corpus corresponding to the audio stream data;

[0117] S4: Determine a target intent corresponding to the audio stream data from cached intent recognition results according to verification results of the first intent corpus and the second intent corpus, where the target intent is used to represent semantics corresponding to the audio stream data.

[0118] Optionally, in this embodiment, the above-mentioned storage medium may include but is not limited to: a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk, and other media that can store program codes.

[0119] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments and optional implementation modes, and this embodiment will not be described in detail here.

[0120] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0121] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A semantic recognition method for audio stream data, characterized in that: include: Acquire a first recognition result obtained when automatic speech recognition (ASR) is performed on audio stream data from a terminal device; When an end mark of a voice activity detection (VAD) threshold appears in the first recognition result, obtaining an audio to be processed corresponding to the end mark of the VAD threshold from the audio stream data, determining a first intent corpus corresponding to the audio to be processed, and caching an intent recognition result of the first intent corpus; After determining that the ASR of the audio stream data is completed, obtaining a second recognition result of the audio stream data, wherein the second recognition result at least includes a second intent corpus corresponding to the audio stream data; According to the verification results of the first intent corpus and the second intent corpus, a target intent corresponding to the audio stream data is determined from the cached intent recognition results, and the target intent is used to represent the semantics corresponding to the audio stream data.

2. The method for semantic recognition of audio stream data according to claim 1, characterized in that: Acquiring a first recognition result obtained when performing automatic speech recognition ASR on audio stream data from a terminal device, including: Performing speech recognition on the audio stream data to obtain speech data and non-speech data; For the voice data, intercepting the voice data based on the start mark of the VAD threshold and the end mark of the VAD threshold to obtain intercepted voice; Perform text recognition on the intercepted speech to obtain the first recognition result.

3. The method for semantic recognition of audio stream data according to claim 1, characterized in that: Determining a target intent corresponding to the audio stream data from cached intent recognition results according to verification results of the first intent corpus and the second intent corpus, including: When it is determined that the verification result indicates that the first intent corpus and the second intent corpus are consistent, obtaining a preprocessing progress of natural language processing on the first intent corpus; When it is determined that the preprocessing progress is completed, determining a first intent obtained by performing natural language processing on the first intent corpus as the target intent; When it is determined that the verification result indicates that the first intent corpus and the second intent corpus are inconsistent, a second intent obtained after natural language processing of the second intent corpus is obtained from the cached intent recognition result, and the second intent is determined as the target intent.

4. The method for semantic recognition of audio stream data according to claim 3, characterized in that: After obtaining the preprocessing progress of natural language processing on the first intention corpus, the method further includes: When it is determined that the preprocessing progress is not completed, continue to monitor the preprocessing progress of the natural language processing of the first intention corpus until the preprocessing progress is completed; Obtain a first processing time for natural language processing of the first intent corpus, and when it is determined that the first processing time is greater than a preset time, obtain a third intent obtained after natural language processing of the second intent corpus, and determine the third intent as the target intent.

5. The method for semantic recognition of audio stream data according to claim 3, characterized in that: After obtaining the preprocessing progress of natural language processing on the first intention corpus, the method further includes: When it is determined that the preprocessing progress is not completed, continue to monitor the preprocessing progress of performing natural language processing on the first intent corpus, and obtain a second processing time of performing natural language processing on the first intent corpus; When it is determined that the second processing duration is less than a preset duration, the first intention is determined as the target intention.

6. The method for semantic recognition of audio stream data according to claim 3, characterized in that: After determining the first intent obtained by performing natural language processing on the first intent corpus as the target intent, the method further includes: Determine the device number of the terminal device, and obtain a device list corresponding to the device number; Determine different terminal devices corresponding to all terminal device numbers in the device list, and select a target terminal device with a target device type from the different terminal devices, wherein the target terminal device is used to send a response message generated based on the target intent to the target object.

7. The method for semantic recognition of audio stream data according to claim 3, characterized in that: After determining the first intent obtained by performing natural language processing on the first intent corpus as the target intent, the method further includes: Respectively obtaining an audio format of the audio to be processed, an audio integrity of the audio to be processed, a signal-to-noise ratio of the audio to be processed, and a frequency spectrum of the audio to be processed; A rejection operation is performed on the audio to be processed based on an audio format of the audio to be processed, an audio integrity of the audio to be processed, a signal-to-noise ratio of the audio to be processed, and a frequency spectrum of the audio to be processed.

8. A semantic recognition device for audio stream data, characterized in that: include: An obtaining module, used to obtain a first recognition result obtained when automatic speech recognition ASR is performed on audio stream data from a terminal device; A first determination module is used to obtain, from the audio stream data, an audio to be processed corresponding to the end identifier of the VAD threshold when an end identifier of a voice activity detection (VAD) threshold appears in the first recognition result, determine a first intent corpus corresponding to the audio to be processed, and cache an intent recognition result of the first intent corpus; an acquisition module, configured to acquire a second recognition result of the audio stream data after determining that the ASR of the audio stream data is completed, wherein the second recognition result at least includes a second intent corpus corresponding to the audio stream data; The second determination module is used to determine the target intent corresponding to the audio stream data from the cached intent recognition results according to the verification results of the first intent corpus and the second intent corpus, and the target intent is used to represent the semantics corresponding to the audio stream data.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 7 when executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 7 through the computer program.

Images