Voice robot interaction method and device based on integrated ASR, medium, program product and terminal

Abstract

According to the voice robot interaction method and device based on the integrated ASR, the medium, the program product and the terminal provided by the invention, rapid deployment and optimization of an intelligent voice interaction system are realized by integrating a mature third-party voice recognition technology and a self-developed voice event model; product deployment and use can be quickly completed when the system enters a new region and a language market, the recognition capability is greatly enhanced, the user interaction experience is optimized, low-delay and smooth and natural interaction is realized, and the adaptability and competitiveness of the product are improved.

Description

Technical Field

[0001] The present application relates to the field of speech processing, and in particular to a speech robot interaction method, device, medium, program product and terminal based on integrated ASR. Background Art

[0002] When entering new regions and language environment markets, when the company's self-developed automatic speech recognition (ASR) technology is not yet mature, it is usually necessary to rely on third-party ASR services to support the rapid development and deployment of the business. For example, in the English-speaking markets in Europe and the United States, third-party ASR services such as Google or Microsoft are used to reduce R&D costs. However, most third-party ASRs currently use a combination of voice activity detection (VAD) and ASR in voice call applications. There are a large number of misjudgment and recognition problems, which may cause certain voice recognition delays. In addition, because the judgment logic of the voice robot is relatively fixed, when the user does not speak temporarily or needs additional information, the robot lacks support, resulting in poor interaction effects.

[0003] Therefore, the current technology has the following defects and shortcomings: lack of good noise resistance, it is difficult to effectively distinguish between speech, noise and background sound, which affects the quality of conversation. At the same time, it lacks intelligent interruption and dynamic fallback functions, and cannot flexibly handle scenarios where users interrupt or operate incorrectly. These problems show that the robot's language generation logic is relatively rigid, resulting in a rigid conversation process and poor interaction effect. Summary of the invention

[0004] In view of the shortcomings of the prior art mentioned above, the purpose of the present application is to provide a voice robot interaction method, device, medium, program product and terminal based on integrated ASR, which is used to solve the problems of existing voice robots lacking good noise resistance and intelligent interruption and fallback functions, resulting in affected conversation quality, rigid production logic and poor interaction effect.

[0005] To achieve the above-mentioned purpose and other related purposes, the first aspect of the present application provides a method, a voice robot interaction method based on integrated ASR, the method being applied to a robot management platform, the method comprising: deploying a third-party ASR engine and a voice event model; when an incoming call is detected, accessing the call and building a conversation task; while the conversation task is ongoing, calling the third-party ASR engine to perform a voice recognition operation to generate a voice recognition text; at the same time, calling the voice event model to perform an event monitoring operation on the conversation task, and generating corresponding voice event information in response to the occurrence of a voice event; generating a dynamic response strategy based on the voice recognition text and the voice event information, and performing corresponding voice interaction operations based on the dynamic response strategy; and automatically closing the conversation task when the call is detected to be over.

[0006] In some embodiments of the first aspect of the present application, the process of calling the voice event model to perform event monitoring operations on the conversation task, and generating corresponding voice event information in response to the occurrence of a voice event includes: continuously monitoring the conversation task, obtaining a real-time audio stream from the conversation task, and performing a noise reduction operation on the real-time audio stream; importing the real-time audio stream after the noise reduction operation into the voice event model; and determining whether the real-time audio stream contains a preset voice event; in response to a preset voice event, determining the event type of the voice event, and generating corresponding voice event information.

[0007] In some embodiments of the first aspect of the present application, the voice event information includes one or more of the following information: event label, timestamp, acoustic feature data, and context information.

[0008] In some embodiments of the first aspect of the present application, the event type includes one or more of the following types: start of utterance, end of utterance, silence, noise, keywords, voice activation, voice input error, voice event rollback, voice event intention, voice event pause, and voice event resumption.

[0009] In some embodiments of the first aspect of the present application, the process of generating a dynamic response strategy based on the speech recognition text and the voice event information includes: contextually splicing the speech recognition text and the voice event information based on a timestamp, and performing natural language processing on the spliced ​​text information to generate an intent recognition result and slot information; and performing a contextual understanding operation on the speech recognition text based on the intent recognition result and the slot information to generate a dynamic response strategy.

[0010] In some embodiments of the first aspect of the present application, when the call is detected to be ended, the following operation is performed after the session task is automatically closed: information is marked on the session task, and the marked data is stored in the management system database of the client.

[0011] To achieve the above-mentioned purpose and other related purposes, the second aspect of the present application provides a voice robot interaction device based on integrated ASR, including: a conversation initialization module: used to deploy a third-party ASR engine and a voice event model; when an incoming call is detected, the call is connected and a conversation task is constructed; a voice processing module: used to call the third-party ASR engine to perform a voice recognition operation during the duration of the conversation task to generate a voice recognition text; at the same time, the voice event model is called to perform an event monitoring operation on the conversation task, and in response to the occurrence of a voice event, corresponding voice event information is generated; an interactive response module: used to generate a dynamic response strategy based on the voice recognition text and the voice event information, and perform corresponding voice interaction operations based on the dynamic response strategy; when the call is detected to be over, the conversation task is automatically closed.

[0012] To achieve the above-mentioned purpose and other related purposes, the third aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the voice robot interaction method based on integrated ASR is implemented.

[0013] To achieve the above-mentioned purpose and other related purposes, the fourth aspect of the present application provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer implements the voice robot interaction method based on integrated ASR.

[0014] To achieve the above-mentioned purpose and other related purposes, the fifth aspect of the present application provides an electronic terminal, including a memory, a processor and a computer program stored in the memory; the processor executes the computer program to implement the integrated ASR-based voice robot interaction method.

[0015] As described above, the voice robot interaction method, device, medium, program product and terminal based on integrated ASR of the present application have the following beneficial effects: When entering new regions and language markets, the present invention can flexibly integrate mature third-party speech recognition (ASR) models at a lower cost to quickly complete product deployment and put it into use. This not only strengthens the recognition capabilities that rely on third-party ASR, but also optimizes the user interaction experience, achieving lower latency and smoother and more natural interactions. Through the dual-engine mode that integrates the speech recognition model and the voice event model, when the self-developed model is not yet mature, the present application can be flexibly configured according to different application scenarios, showing a high reuse value. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flow chart of an embodiment of a voice robot interaction method based on integrated ASR of the present application is shown.

[0017] Figure 2 A structural schematic diagram of an embodiment of a voice robot interaction device based on integrated ASR of the present application is shown.

[0018] Figure 3 A structural diagram of an embodiment of a voice robot interaction terminal based on integrated ASR of the present application is shown. DETAILED DESCRIPTION

[0019] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0020] Before further describing the present invention in detail, the nouns and terms involved in the embodiments of the present invention are explained. The nouns and terms involved in the embodiments of the present invention are applicable to the following interpretations:

[0021] <1> ASR (Automatic Speech Recognition): Automatic speech recognition technology that can convert speech into text. These technologies use machine learning algorithms to recognize and transcribe speech signals to achieve speech-to-text conversion.

[0022] <2> Voice robots: robots that can interact with humans through voice. They usually use ASR technology to recognize the user's voice input and use text-to-speech synthesis technology to generate voice responses.

[0023] <3> Bot management platforms: Platforms used to manage and deploy bots. These platforms provide a central console that allows developers and administrators to create, configure, deploy, and monitor bots. These platforms typically support multiple types of bots, including voice bots.

[0024] <4> Third-party ASR engines: ASR technologies provided by third parties. These engines can be integrated into other applications or platforms to provide speech recognition capabilities. Third-party ASR engines can provide higher accuracy and better support, especially in specific domains or languages.

[0025] <5> Voice event model: A model used to describe and process voice events. Voice event models define the structure and semantics of voice events, including event types, event parameters, and event processing logic. They can be used in voice robots and other voice applications.

[0026] <6> Voice events: events in voice interaction, such as a user saying a sentence or a voice robot responding to a message. Voice events can trigger specific actions or processing logic, such as executing a command or answering a question.

[0027] To facilitate understanding of the embodiments of the present application, first Figure 1 Detailed description. Figure 1 The flowchart of a voice robot interaction method based on integrated ASR in an embodiment of the present invention is shown. The voice robot interaction method based on integrated ASR in this embodiment mainly includes the following steps:

[0028] Step S11: deploy a third-party ASR engine and a voice event model; when an incoming call is detected, connect to the call and build a conversation task.

[0029] In one embodiment of the present application, the Robot Management Platform (RM) is a software solution for managing and monitoring the robot system. It is used to manage and monitor the robot system and has the functions of real-time equipment monitoring, automatic task scheduling, data analysis, remote control, user authority management, and interface integration with other systems.

[0030] In one embodiment of the present application, the third-party ASR engine (ASR, Automatic Speech Recognition) refers to a speech recognition service provided by a third-party company, which converts human speech signals into text. It analyzes the sound features in the audio signal, recognizes the words and syllables in the speech, and converts them into readable text. The third-party ASR engine includes but is not limited to: Google Cloud Speech-to-Text, IBM Watson Speech to Text, Microsoft Azure Speech Service, Amazon Transcribe, iFlytek, etc.

[0031] In one embodiment of the present application, the Speech Event Detection (SED) model is a technology for identifying and classifying specific speech events, including event recognition, time tagging, classification capabilities, and context understanding.

[0032] In one embodiment of the present application, the voice event model includes a self-developed training model, and the voice event detection of the self-developed training model includes the following steps for determining the start and end of a conversation: First, a VAD algorithm is used to make a preliminary judgment on the speech segment to be judged. If the VAD judgment result is that the speech has started, it is directly confirmed; if the judgment result is that the speech has not started and the current speech segment has not ended, the secondary judgment stage is entered. In the secondary judgment, a set of relevant speech frames is extracted, and the ASR algorithm is used for re-judgment, and whether the speech has started is determined by the number of characters in the recognition result. For the specific judgment of starting to speak, the model sets a continuous timing start frame, calculates the speech duration, and compares it with the standard duration (such as 300ms). At the same time, in order to improve accuracy, the speech frame segments that do not meet the standard tail misjudgment duration (such as 180ms) will be discarded. The judgment of ending the speech is determined by setting the pause timing start frame, calculating the pause time, and comparing it with the standard pause time (such as 500ms). If the pause time meets the standard and there is no speech in the tail frame, it is determined to end the speech. In the process of voice event detection, short voices that occur during pauses are discarded to avoid misjudgment. The voice event detection model combines VAD and ASR algorithms and sets multiple duration thresholds. Through multiple judgments and optimization strategies, the accuracy of voice event detection is effectively improved, and the accuracy of conversation start and conversation end detection is improved.

[0033] Furthermore, a method for determining voice interruption in a conversation is also provided in the voice event detection model. Specifically, a large number of human-computer conversation recording files are first collected and preprocessed, and then the shortest duration of the voice when the person starts speaking when the robot plays the voice is counted and classified by user type. Based on the statistical results, voice interruption judgment rules are formulated, including determining the shortest duration of the voice for interrupting the conversation. In real-time conversations, VAD technology is used to process the user voice stream in real time, obtain the duration of continuous voice, and apply judgment rules to make voice interruption judgments. According to the judgment results, the conversation strategy is adjusted, such as stopping the robot voice or continuing the normal conversation. Finally, new data is regularly collected to update the statistical results and judgment rules, and the model performance is continuously optimized. The entire process forms a closed loop, and each step is closely related to ensure the accuracy and practicality of voice interruption judgments.

[0034] In one embodiment of the present invention, the training method of the voice event detection model includes: first obtaining the voice data to be recognized, and then inputting it into a voice recognition integrated model including two models. The voice recognition integrated model includes a first voice recognition model and a second voice recognition model. Among them, the first model has fewer parameters and faster operation speed, and is used to quickly determine whether there is a human voice and its boundary in the voice data to be recognized. The first model adopts a shallower neural network structure, which is used to process audio features in a streaming mode, and perform real-time recognition and feedback, which significantly reduces the delay of the system. The second voice recognition model is relatively complex and has a large number of parameters, and is used to perform more accurate voice recognition on the human voice part recognized by the first model. The model adopts a deeper neural network architecture, which can capture richer voice features, thereby improving the accuracy and robustness of recognition. Working in a non-streaming mode, it can perform in-depth analysis of the complete voice data, which is suitable for application scenarios requiring high accuracy. During the processing process, the integrated model is divided into streaming and non-streaming according to the working mode. The streaming mode allows real-time input and gradual return of recognition results, which is suitable for applications that require fast response, such as voice assistants and real-time translation. The non-streaming mode waits for the user to complete the voice input before starting the recognition process, which is suitable for high-precision recognition of longer speech.

[0035] Furthermore, in non-streaming mode, the first model recognizes the human voice part and extracts the timestamp, and then the second model performs accurate recognition; in streaming mode, the first model determines in real time whether there is a human voice, and once detected, it is synchronously input into the second model for recognition. Finally, the integrated model outputs the corresponding text information.

[0036] Preferably, the voice event detection model also includes an optimization strategy, including but not limited to: using fixed-length data segments, using fine-grained modeling units to train the first model, and moving the recognition starting point forward by statistical analysis in streaming mode. Among them, the process of using fixed-length data segments includes: dividing the long audio data to be recognized into several fixed-length data segments (for example, each segment is 10 seconds) for recognition processing to reduce memory consumption and speed up recognition. Using fine-grained modeling units to train the first model means that when training the first speech recognition model, fine-grained modeling units are used, such as decomposing Chinese characters into single words or decomposing English words into letters, to improve the matching and robustness of the model to speech signals. The process of moving the recognition starting point forward by statistical analysis in streaming mode includes: the first speech recognition model moves the recognition starting point forward by statistical analysis methods. Specifically, the model records the error between the actual start time of the human voice and the recognition time in the test phase, and adjusts the recognition starting point according to this error to ensure that the front end of the human voice is not missed during the recognition process, thereby improving the accuracy and completeness of the recognition.

[0037] In one embodiment of the present application, the voice event model supports the creation of conversation tasks, and determines the start and end time of speaking by detecting voice events, and supports multi-model event recognition and determination, such as speaking, silence, noise, modal particle optimization and keyword optimization. According to specific market needs and business environments, relevant corpus information can be collected, such as customer call record data or third-party open source corpus information. By performing feature classification on these corpus information, modal particles, background noise, etc. are marked as specific events respectively, and then the accuracy of speech recognition is improved through unsupervised training. In this way, low-latency speech start / end judgment and real-time response are achieved, and the experience of human-computer interaction is enhanced. The voice event types in this embodiment are shown in Table 1. The voice event types included therein are: conversation start, start speaking, stop speaking, conversation end, speech occurs, silence, noise, and keywords.

[0038]

[0039] Table 1: Voice event types

[0040] In one embodiment of the present application, the method of deploying the third-party ASR engine and voice event model includes: calling the third-party ASR engine and voice event model through an API, sending audio data to the third-party ASR engine or voice event model through an API interface such as RESTful or WebSocket, and obtaining real-time recognition results.

[0041] In one embodiment of the present application, when the call is connected, RM will immediately establish an independent session task S1 (after the call is hung up, the current session task S1 ends). At this time, RM synchronously calls the integrated ASR dual-engine resources to perform the speech recognition task of S1. The third-party ASR engine and the voice event model dual engine will process speech recognition in parallel, and synchronize the transcribed text information to the session task S1 of the RM platform in real time, while marking a timestamp for each message.

[0042] Step S12: During the duration of the conversation task, the third-party ASR engine is called to perform speech recognition operations to generate speech recognition text; at the same time, the speech event model is called to perform event monitoring operations on the conversation task, and corresponding speech event information is generated in response to the occurrence of speech events.

[0043] In one embodiment of the present application, the process of calling the voice event model to perform event monitoring operations on the conversation task and generating corresponding voice event information in response to the occurrence of a voice event includes: continuously monitoring the conversation task, obtaining a real-time audio stream from the conversation task, and performing a noise reduction operation on the real-time audio stream; importing the real-time audio stream after the noise reduction operation into the voice event model; and determining whether the real-time audio stream contains a preset voice event; in response to a preset voice event, determining the event type of the voice event and generating corresponding voice event information.

[0044] In an embodiment of the present application, the voice event information includes one or more of the following information: event label, timestamp, acoustic feature data, and context information.

[0045] In one embodiment of the present application, the event type includes one or more of the following types: start of utterance, end of utterance, silence, noise, keyword, voice activation, voice input error, voice event rollback, voice event intention, voice event pause, and voice event resumption.

[0046] In one embodiment of the present application, by default, the text output by the third-party ASR engine is regarded as the first NLU text. The text recognized by the speech event model is used to determine the state of the S1 call process, including the start / end of speech, silence, noise, and recognition of keywords. To better fit the actual application scenario, when the ASR event recognizes the optimized and enhanced modal particles and keywords, the RM will automatically switch the ASR event recognition text to NLU text.

[0047] In one embodiment of the present application, the process of the speech event monitoring model performing event monitoring operations on the conversation task includes: the speech event monitoring model is based on a deep learning model, such as a reinforcement learning framework, to train corpus features. In this process, the model learns to recognize specific modal particles, pauses, and speech start features through a large amount of annotated data. This training method enhances the model's ability to perceive the user's voice, enabling it to clearly and effectively determine when the user starts speaking in a complex noisy environment.

[0048] Furthermore, when the voice event monitoring model detects background noise in the environment (such as music, mechanical sounds, etc.), the system can effectively identify it as a non-recognizable voice signal through spectrum analysis. In an exemplary manner, based on a variety of feature extraction technologies such as Short-Time Fourier Transform (STFT) and Mel-Frequency Cepstral Coefficients (MFCC), the exclusion rate of non-voice signals is improved.

[0049] In one embodiment of the present application, the process of performing noise reduction on the real-time audio stream includes: using a dynamic threshold strategy to automatically adjust the model sensitivity according to the change of real-time environmental noise. This enables a high recognition rate to be maintained in different audio environments, ensuring accurate judgment of the start of user speech.

[0050] In one embodiment of the present application, during a call, the RM platform autonomously adjusts the preset parameters of the voice event monitoring model according to the set application scenario. These preset parameters include: high value and low value. The high value means that when the ASR event recognizes that the user starts speaking, the interruption is performed immediately; the low value means that when the ASR event recognizes the keywords on the user side, the interruption is performed. Once the interruption is performed, the RM platform will immediately stop the work of the current broadcast robot.

[0051] Furthermore, after stopping the current broadcast robot, the robot is dynamically reset, and in the conversation task S1, the system completes the context splicing of the recognized text and the newly recognized text based on the timestamp to generate a new intent. After that, the system will re-process the spliced ​​text with natural language understanding (NLU). After judging that the speech is over, the system will generate new broadcast content based on the newly recognized intent. This process significantly improves the ability and accuracy of human-computer interaction.

[0052] In one embodiment of the present application, during the execution of a conversation task, the third-party ASR engine and the voice event model use streaming voice recognition, and the third-party ASR engine can generate text results in real time and feed them back to the RM platform, thereby providing users with instant recognition feedback, ensuring rapid response to user input, and avoiding delays. At the same time, the system uses an event response mechanism to automatically adjust the dialogue state and update the user interface when a voice event (such as the start or end of speaking) is recognized to meet real-time interaction needs. The generation and processing of event information adopts an asynchronous mechanism, which effectively reduces system congestion caused by event detection and achieves a smooth user experience.

[0053] Step S13: Generate a dynamic response strategy based on the speech recognition text and the speech event information, and perform a corresponding speech interaction operation based on the dynamic response strategy.

[0054] In one embodiment of the present application, the process of generating a dynamic response strategy based on the speech recognition text and the voice event information includes: contextually splicing the speech recognition text and the voice event information based on a timestamp, and performing natural language processing on the spliced ​​text information to generate an intent recognition result and slot information; and performing a contextual understanding operation on the speech recognition text based on the intent recognition result and the slot information to generate a dynamic response strategy.

[0055] In one embodiment of the present application, the process of generating a dynamic response strategy based on the speech recognition text and the speech event information includes: intent recognition, context management, strategy generation model and personalized response. First, multiple intent categories are defined in RM, such as querying information, setting reminders, controlling devices, chatting, etc. Classification algorithms (such as SVM, deep learning classifiers) are used to identify user intent. Intent understanding is performed in combination with context information, and the context window is used to take into account the most recent interaction records. Secondly, a context management module is designed to maintain the state of the user's current session (such as the previous instruction, the topic of the conversation, etc.), helping the system to consider the continuity of the context when generating a response. Through conversation tracking technology, the conversation flow in multiple user conversations is tracked to improve the coherence of the interaction. Furthermore, a generative model (such as the GPT series) or a template-based system is used to generate natural responses based on the user's intent and context. Using multi-round dialogue strategies, a series of possible responses are formulated, and the optimal path is selected based on user feedback and behavior. Finally, the response content is adjusted according to the user's historical interaction records and preferences. For example, if the user often inquires about the weather, the system can prioritize weather information. Utilize collaborative filtering recommendation algorithms, combined with social input, to provide users with personalized recommendations and enhance user experience.

[0056] In one embodiment of the present application, after executing the corresponding voice interaction operation based on the dynamic response strategy, the following operations are also performed: construct a real-time feedback mechanism, use the user's selection, rating and text feedback data, and combine incremental learning algorithms (such as online gradient descent) to perform online updates of the model to ensure that the system can quickly adapt to changes in user needs. By introducing user satisfaction evaluation indicators (such as Net Promoter Score), the system can quantitatively feedback user satisfaction, while sentiment analysis tools (such as VADER or TextBlob) can analyze the emotional tendency of the interactive text and accurately capture the user's emotional response. In addition, the system also establishes a real-time monitoring architecture, uses data flow technology (such as Apache Kafka) to process and analyze interactive data in real time, and identifies key performance indicators (KPIs) such as response time and success rate to monitor changes in system performance and user behavior patterns. Based on these data, the system can automatically update the response model and strategy library regularly, and use technologies such as transfer learning to combine new data with pre-trained models, thereby ensuring flexibility in responding to new user needs and environmental changes to improve the intelligence level of voice interaction.

[0057] Step S14: When the call is detected to be over, the conversation task is automatically closed.

[0058] In one embodiment of the present application, when the call is detected to be over, the following operation is performed after the session task is automatically closed: information is marked on the session task, and the marked data is stored in the management system database of the client.

[0059] In one embodiment of the present application, when the call is detected to be over, the following operations are performed after the session task is automatically closed: First, natural language processing is performed on the call content to extract keywords and topics, and the user's emotional state is analyzed to provide data support for subsequent customer service optimization. Second, multi-dimensional tags are generated based on the call content, and the multi-dimensional tags include but are not limited to: call topics, keywords, conflict detection, call duration, number of transfers, etc. Third, in order to improve the accuracy of automatic labeling of multi-dimensional tags, adaptive learning and optimization are performed based on historical call data.

[0060] Furthermore, the generated tag data is stored in the client system to achieve synchronous sharing across systems and enhance the overall data value. In addition, the system also supports real-time analysis and dynamic report generation to help managers understand key customer service issues and satisfaction trends in a timely manner, so as to adjust strategies. Finally, in the process of data management, encryption and desensitization measures are also taken to ensure customer privacy and compliance.

[0061] It should be noted that in the embodiments of the present application, words such as "exemplary" or "for example" represent examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0062] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can represent: a, b, c, ab, ac, bc or abc, where a, b, c can be single or multiple.

[0063] Figure 2 is a schematic block diagram of a voice robot interaction device based on integrated ASR provided in an embodiment of the present application. Figure 2 As shown, the device includes a conversation initialization module, a voice processing module and an interactive response module.

[0064] The session initialization module 201 is used to deploy a third-party ASR engine and a voice event model; when an incoming call is detected, the call is connected and a session task is constructed.

[0065] Speech processing module 202: used to call the third-party ASR engine to perform speech recognition operations to generate speech recognition text during the ongoing process of the conversation task; at the same time, call the speech event model to perform event monitoring operations on the conversation task, and generate corresponding speech event information in response to the occurrence of speech events.

[0066] Interaction response module 203: used to generate a dynamic response strategy based on the voice recognition text and the voice event information, and perform corresponding voice interaction operations based on the dynamic response strategy; when the call is detected to be over, the conversation task is automatically closed.

[0067] It should be understood that the specific process of each module executing the above corresponding steps has been described in detail in the above method embodiment, and for the sake of brevity, it will not be repeated here.

[0068] It should also be understood that the division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present application may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0069] Figure 3 is a schematic block diagram of an electronic terminal provided in an embodiment of the present application. Figure 3 As shown, the electronic terminal includes: at least one processor 301, a memory 302, at least one network interface 303 and a user interface 305. The various components in the device are coupled together through a bus system 304. It can be understood that the bus system 304 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 304 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 3 In the specification, various buses are labeled as bus systems.

[0070] The user interface 305 may include a display, a keyboard, a mouse, a trackball, a click gun, keys, buttons, a touch pad or a touch screen.

[0071] It is understood that the memory 302 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable categories of memory.

[0072] The memory 302 in the embodiment of the present invention is used to store various types of data to support the operation of the electronic terminal 300. Examples of these data include: any executable program used to operate on the electronic terminal 300, such as an operating system 3021 and an application 3022; the operating system 3021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application 3022 can include various applications, such as a media player (Media Player), a browser (Browser), etc., for implementing various application services. The method provided by the embodiment of the present invention can be included in the application 3022.

[0073] The method disclosed in the above embodiment of the present invention can be applied to the processor 301, or implemented by the processor 301. The processor 301 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 301 or the instruction in the form of software. The above processor 301 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 301 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general processor 301 can be a microprocessor or any conventional processor, etc. In combination with the steps of the accessory optimization method provided in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0074] In an exemplary embodiment, the electronic terminal 300 may be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD) to execute the aforementioned method.

[0075] According to the method provided in the embodiment of the present application, the present application also provides a computer program product, which includes: a computer program code, when the computer program code is run on a computer, the computer executes Figure 1 The above method of any one of the embodiments shown in the embodiments.

[0076] According to the method provided in the embodiment of the present application, the present application also provides a computer-readable storage medium, which stores a program code, and when the program code is run on a computer, the computer executes Figure 1 The above method of any one of the embodiments shown in the embodiments.

[0077] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program and / or a computer. By way of illustration, both applications running on a computing device and a computing device can be components. One or more components may reside in a process and / or an execution thread, and a component may be located on a computer and / or distributed between two or more computers. In addition, these components may be executed from various computer-readable media having various data structures stored thereon. Components may, for example, communicate through local and / or remote processes according to signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system and / or a network, such as the Internet interacting with other systems through signals).

[0078] Those of ordinary skill in the art will appreciate that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0079] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0080] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0081] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0082] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0083] In the above embodiments, the functions of each functional unit can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When loading and executing computer program instructions (programs) on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (Digital Subscriber Line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server, data center, etc. that contains one or more available media integrated. Available media may be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., high-density digital video discs (DVDs), or semiconductor media (e.g., solid state disks (SSDs)).

[0084] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0085] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0086] In summary, the present application provides a voice robot interaction method, device, medium, program product and terminal based on integrated ASR. The present invention provides an integrated voice method based on a third-party voice recognition engine and a voice event model. By integrating mature third-party voice recognition technology and self-developed voice event models, the rapid deployment and optimization of intelligent voice interaction systems can be achieved. When entering new regions and language markets, product deployment can be quickly completed and put into use, which greatly enhances recognition capabilities, optimizes user interaction experience, achieves low latency and smooth and natural interaction, and improves product adaptability and competitiveness. Therefore, the present application effectively overcomes the various shortcomings of the prior art and has a high industrial utilization value.

[0087] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A voice robot interaction method based on integrated ASR, characterized in that: The method is applied to a robot management platform, and the method comprises: Deploy a third-party ASR engine and voice event model; when an incoming call is detected, connect to the call and build a conversation task; During the ongoing process of the conversation task, the third-party ASR engine is called to perform a speech recognition operation to generate a speech recognition text; at the same time, the speech event model is called to perform an event monitoring operation on the conversation task, and in response to the occurrence of a speech event, corresponding speech event information is generated; Generate a dynamic response strategy based on the speech recognition text and the speech event information, and perform a corresponding speech interaction operation based on the dynamic response strategy; When the call is detected to be over, the conversation task is automatically closed.

2. The voice robot interaction method based on integrated ASR according to claim 1 is characterized in that: The process of calling the voice event model to perform event monitoring operation on the conversation task and generating corresponding voice event information in response to the occurrence of the voice event includes: Continuously monitoring the conversation task, obtaining a real-time audio stream from the conversation task, and performing a noise reduction operation on the real-time audio stream; Importing the real-time audio stream after the noise reduction operation into the voice event model; and determining whether the real-time audio stream contains a preset voice event; In response to a preset voice event, an event type of the voice event is determined, and corresponding voice event information is generated.

3. The voice robot interaction method based on integrated ASR according to claim 2 is characterized in that: The speech event information includes one or more of the following information: event label, timestamp, acoustic feature data and context information.

4. The voice robot interaction method based on integrated ASR according to claim 2 is characterized in that: The event types include one or more of the following types: utterance start, utterance end, silence, noise, keyword, voice activation, voice input error, voice event rollback, voice event intention, voice event pause, and voice event resumption.

5. The voice robot interaction method based on integrated ASR according to claim 1, characterized in that: The process of generating a dynamic response strategy based on the speech recognition text and the speech event information includes: Performing context splicing on the speech recognition text and the speech event information based on the timestamp, and performing natural language processing on the spliced ​​text information to generate an intent recognition result and slot information; Based on the intention recognition result and the slot information, a context understanding operation is performed on the speech recognition text to generate a dynamic response strategy.

6. The voice robot interaction method based on integrated ASR according to claim 1, characterized in that: When the call is detected to be over, the following operations are performed after the conversation task is automatically closed: information is marked on the conversation task, and the marked data is stored in the management system database of the client.

7. A voice robot interaction device based on integrated ASR, characterized in that: include: Conversation initialization module: used to deploy third-party ASR engines and voice event models; when an incoming call is detected, the call is connected and a conversation task is constructed; The speech processing module is used to call the third-party ASR engine to perform speech recognition operations to generate speech recognition text during the ongoing process of the conversation task; at the same time, call the speech event model to perform event monitoring operations on the conversation task, and generate corresponding speech event information in response to the occurrence of speech events; Interaction response module: used to generate a dynamic response strategy based on the voice recognition text and the voice event information, and perform corresponding voice interaction operations based on the dynamic response strategy; when the call is detected to be over, the conversation task is automatically closed.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the voice robot interaction method based on integrated ASR as described in any one of claims 1 to 6 is implemented.

9. A computer program product, characterized in that The computer program product includes computer program code, and when the computer program code is executed on a computer, the computer implements the voice robot interaction method based on integrated ASR as claimed in any one of claims 1 to 6.

10. An electronic terminal comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the voice robot interaction method based on integrated ASR as described in any one of claims 1 to 6.

Images