System and method for optimizing user interaction session in interactive voice response system

Abstract

To provide a system and a method for monitoring and optimizing a user interaction session in an interactive voice response system during a human-computer interaction.SOLUTION: A user interaction management system (100) for monitoring and optimizing a user interaction session includes a conversation controller module (109). The conversation controller module (109) receives a voice feature during the user interaction session, processes output of ASR (automatic speech recognition) and NLU (natural language understanding) and optimizes duration time of the user interaction session. A conversation controller may suggest a TTS (text to speech) module (107) to increase or decrease the rate of speech.SELECTED DRAWING: Figure 1A

Description

[Technical field]

[0001] The present invention relates to monitoring and control of conversations and user interaction sessions in interactive voice response systems, and more particularly to a system and method for improving user interaction by optimizing the duration of interaction sessions within an interactive voice response system. [Background technology]

[0002] In general, a user journey is described as the various steps a user takes to complete a particular task within a system, application, or website. In the case of a dialog engine, the user journey is defined as the time span from when a user (sentence or input) starts a conversation and invokes a service to when the service is completed (when the dialog engine finishes announcing that the desired service has been provided successfully). However, in a user interaction session with an interactive voice response system using a dialog engine, the user journey duration is affected by several phenomena, such as silence intervals and time involved in predicting turn transitions, the text-to-speech (TTS) process, the execution of natural language understanding (NLU) models, and conversation route planning. The inconsistency of several phenomena wastes the user's time and energy, and also incurs system time, ASR and server overload, and operational costs, respectively. As a result, users may find the system interface to be unfriendly and may encounter difficulties when entering long information such as phone numbers due to the lack of a better silence detection mechanism, which may discourage them from using the system interface in the future. Moreover, in the current situation, in the dialogue management system of IVR communication systems, the ASR and natural language understanding (NLU) pipeline of the user interface driven entities needs to be manually adapted and executed for each classified case, which makes large-scale maintenance difficult, resulting in multiple cases being overlooked or poorly navigated, impairing the progress of the conversation.

[0003] Optimizing at least one phenomenon can optimize the duration of the user journey. For example, if the number of turn transitions is optimized, the number of times that the ASR model needs to infer speech is reduced, and the number of times that the TTS model needs to convert text to speech is similarly reduced. This can reduce the size of the server required to serve multiple concurrent users, and can also increase the number of users that can be served simultaneously via configuration.

[0004] Therefore, there is a need for an improved solution that is flexible, allows improving and testing the user journey, and allows errors to be recognized and changed in the background without interrupting the user interaction session, and further saves operational costs and user usage costs. Summary of the Invention

[0005] This Summary is provided to introduce some concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0006] Generally, in conventional dialogue management systems, multiple text analysis tools, tuning tools, and optimization tools are applied and operated manually separately, which leads to problems of complexity and cost. Unprecedented cases and misalignments are identified based on human feedback, and countermeasures are taken manually accordingly. Misalignments in dialogue sessions consume time and energy for users. As a result, users may not find the system interface easy to use, and may encounter difficulties when entering long information, such as telephone numbers. To overcome the above related problems, the present invention describes a system and method for improving user dialogue sessions through optimizing the duration of dialogue sessions in an interactive voice response system by determining errors and deploying changes to efficiently develop and optimize and maintain the user journey. According to an embodiment of the present invention, a conversation controller is used that can automatically trigger actions based on results obtained from a session monitoring module and a user profile database, thereby optimizing the user journey and its duration. The system and method for user dialogue session management further includes applying sentiment analysis that focuses on specific components and operates accordingly, thereby improving the duration and experience of the user journey and measuring and reducing uncertainty, classification errors, and misalignments in dialogue sessions in the system. This makes the interface more efficient and familiar, with the added benefit of saving time and costs for system operation and user use. Implementations may have one or more of the following features:

[0007] The figures depict one or more embodiments by way of example only, and not by way of limitation, in which like reference numbers indicate identical or similar elements. [Brief description of the drawings]

[0008] [Figure 1A]FIG. 1A is a block diagram illustrating data sent and received during an interaction session between a user and an IVR communication system 100 for monitoring and optimizing the user interaction session using a conversation controller. [Figure 1B] FIG. 1B is a block diagram illustrating data sent and received during a conversation session between a user and an IVR communication system 120 for monitoring and optimizing a user interaction session using a conversation controller. [Diagram 2] FIG. 2 is a flow chart illustrating a process 200 for determining speech and / or non-speech segments. [Diagram 3] FIG. 3 is a flow chart illustrating a process 300 for conversation controller integration in an example scenario. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Described herein are methods and systems for monitoring and optimizing user interaction sessions in an interactive voice response system during human-computer interaction. The systems and methods are described with reference to figures, which are intended to be illustrative and not limiting, for ease of explanation of exemplary systems and methods according to embodiments of the present invention.

[0010] The foregoing description of specific embodiments is intended to clarify the general nature of the embodiments herein, such that others, by applying current knowledge, may easily modify and / or adapt such specific embodiments to various applications without departing from the general concept, and therefore such applications and modifications should be understood and are intended to be within the scope and equivalents of the meaning of the disclosed embodiments.

[0011] It is also noted that the embodiments may be described as a process that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart describes operations as a series of processes, many operations may be performed simultaneously in parallel. Additionally, the order of operations may be rearranged. A process terminates when an operation is completed, but may have additional steps not included in the diagram. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination corresponds to a return of the function to the calling function or the main function.

[0012] It is to be understood that the phraseology and terminology used herein is for the purpose of description and not of limitation. Thus, while the embodiments herein have been described in terms of preferred embodiments, those skilled in the art will recognize that the embodiments herein can be practiced with modification within the spirit and scope of the appended claims.

[0013] As used herein, the term "network" refers to any form of communication network used to transmit data and connect communication devices (e.g., phones, smartphones, computers, servers) to one another. According to embodiments of the present invention, data includes at least one of processed data and unprocessed data. Such data includes data obtained through automated data processing, manual data processing, and unprocessed data.

[0014] As used herein, the term "artificial intelligence" refers to a set of executable instructions stored on a server and generated using machine learning techniques.

[0015] In the following description, terms such as "first" and "second" are used to describe various elements, but these elements should not be limited by the terms. These terms are used only to distinguish one element from another. For example, a first intent may be expressed as a second intent, and a second intent may be expressed as a first intent, without departing from the scope of the various examples described.

[0016] FIG. 1 is a conceptual diagram illustrating an example framework for the overall data flow between a human and an example interactive voice response (IVR) communication system 100 for monitoring and optimizing a user interaction session according to one or more aspects of the present invention. In this example, the system disclosed herein comprises a two-way audio connector unit 103, a speech voice processing unit 104, a dialogue engine 105, a dialogue engine dispatcher 106, a text-to-speech (hereafter TTS) module 107, a session monitoring module 108, a conversation controller module 109, a user profile database 110, and a user identification module 111. The audio connector unit 103 comprises a voice activity detector (hereafter VAD) module 103a and a speech turn detector module 103b. The conversation voice processing unit 104 further comprises an ASR engine 104a, an ASR model database 104b, an emotion sentiment recognition module 104c, a noise profile and environmental noise classification module 104d, and a voice biometrics module 104e. The dialogue engine 105 further comprises an NLU component 105a, an NLU model database 105b, a dialogue engine core model database 105c, a dialogue engine component and action server module 105d, and a dialogue state tracker module 105e. The TTS module 107 further comprises a TTS model storage 107a and a TTS parameter database 107b. The dialogue engine (105) receives and processes the conversation data and the corresponding edited text, and stores the corresponding dialogue engine components and NLU (Natural Language Understanding) models for processing voice-based interactions with a user during a user interaction session.

[0017] It should be understood, however, that other components not shown may be included, and that some of the components shown may not be present in every device in which aspects of the disclosure may be employed. Further, some components shown as single components may appear multiple times on a single device.

[0018] According to an exemplary embodiment of the present invention, a user 101 initiates a call to the IVR communication system 100 using a client device. For simplicity, the client device is not illustrated herein. The client device may correspond to a wide variety of electronic devices. According to an exemplary embodiment of the present invention, the client device is any telecommunication device such as a smart phone, a feature phone, or a regular landline phone. The client device serves as a service request means for inputting a user's request.

[0019] According to yet another embodiment of the present invention, a user 101 from a client device communicates to the IVR communication system 100 using an application on a smartphone using data services that may or may not use a telecommunications network.

[0020] Returning to FIG. 1 , a call from a user 101 is communicated to the IVR communication system 100 through a telecommunication channel 102. The telecommunication channel 102 routes the call to a two-way audio connector unit 103. The two-way audio connector unit 103 connects to a user identification module 111 for authenticating a user such as the user 101 to the IVR communication system 100. The user identification module 111 connects and registers the user to the IVR communication system 100, and verifies and retains the identity information. The user identification module 111 further stores information about the user in a corresponding user profile in a user profile database 110 to provide multiple personalized services. The user identification module 111 identifies and / or registers the user 101 using the caller number of the user 101 and / or a unique identification number assigned to the user 101 when the user 101 was previously registered. The user identification module 111 is further configured to distinguish the synthetic voice and the human voice of the user in the received conversation data from the stored voice biometrics of the user to detect any fraudulent activity. The user identification module 111 is further configured to store and update a user profile having past and current dialogue session states, call session statistics, ASR models, dialogue engine models, and TTS models corresponding to the existing user profile.

[0021] According to yet another embodiment of the present invention, the user identification module 111 is further configured to launch the conversation controller module 109 upon the user 101 successfully authenticating into the IVR communication system 100 .

[0022] Returning to FIG. 1 , the conversation controller (109) can initiate actions based on the obtained analysis results to perform actions to satisfy the desired intent. The conversation controller module 109 retrieves a plurality of user data of the user 101 from the corresponding user profile in the user profile database 110 when a user interaction session is initiated. The user profile database 110 stores information related to the user, such as past and current conversation statistics and user preferences, such as, but not limited to, ASR models, NLU models, dialogue engine models, and TTS models that are optimal for the user. The user profile database 110 further stores past and current call statistics data corresponding to the user interaction session received from the conversation controller module 109. The conversation controller module (109) receives the audio features and selects and / or modifies ASR and NLU models associated with the user interaction session to optimize the duration of the user interaction session. The conversation controller (109) is further configured to select and modify models associated with the determination of speech segments, speech segments, turn-taking speech segments, and interruption speech segments to achieve a targeted service. The forms and / or slots are populated with service-oriented information received from the user to achieve goals and / or targeted services. A conversation controller module (109) receives the speech features and processes the ASR and NLU output to optimize the interaction session time with the user. According to an embodiment of the present invention, the conversation controller may suggest to the TTS to increase the speech rate. According to yet another embodiment, the conversation controller may suggest to the TTS to decrease the speech rate.

[0023] Additionally, receiving and analyzing the conversational data and voice characteristics from the user's speech input may further include authenticating the user to the user interaction session using the user's caller number or a unique identification number assigned to the user.

[0024] The conversation controller module 109 is further configured to update a user profile corresponding to each user during every interaction session with the user and communicate the data accordingly for optimizing the user journey. The conversation controller module 109 is further configured to assign and / or modify a number of thresholds for determining speech and non-speech segments in the received audio signal corresponding to the speech input of the user 101, such as a non-speech detection threshold, a final non-speech detection threshold, a non-speech duration threshold, an activation threshold for detecting the start of speech, a deactivation threshold for detecting the end of speech, and a user timeout threshold.

[0025] The conversation controller module 109 further updates and / or modifies a number of user attributes corresponding to the profile of the user 101 in the user profile database 110 after each user interaction session, including, but not limited to, the user's expertise level, the user's speaking rate, the duration of the non-speech detection window, the preferred set of conversation path options, the breakdown length of the long information conversation, the model selection from the corresponding ASR, dialogue engine, NLU, TTS, and the average negative sentiment, emotion score, happiness index of the user. The speech processing unit (104) is further configured to detect and analyze at least one of emotion, sentiment, noise profile, and environmental speech information from the features of the received speech data.

[0026] According to a non-limiting example scenario for one embodiment of the present invention, if it is determined, after following analysis of past and current user interactions, that the user 101 often faces difficulties in completing speech input within a turn window, the conversation controller module 109 may increase the non-speech detection window duration threshold. The non-speech detection window duration threshold may be changed both during or after the corresponding user interaction session. Furthermore, the non-speech threshold preferences for each user are stored in the corresponding user profile in the user profile database 110.

[0027] Returning to FIG. 1, the user profile database 110 further stores a number of identifiers for users, such as the user 101. The number of identifiers may include, for example, but not limited to, a user ID, a user name, and other self-identification information of the user, including one or more parameters, such as, for example, but not limited to, a personal identification number, a social security number, and / or a voice biometric voiceprint. The user profile database 110 may also store user demographic information corresponding to each user, such as the user's age, gender, occupation, educational information (e.g., education level, degree, etc.), or one or more locations associated with the user (e.g., the user's home, the user's place of work, a place frequently visited by the user, etc.). The speech voice processing unit 104 is further configured to utilize voice biometrics to identify and register users participating in the user interaction session using the voice biometrics module 104e. Furthermore, authenticating a user to a user interaction session using the user's caller number or a unique identification number assigned to the user further includes utilizing voice biometrics to identify and register users participating in the user interaction session.

[0028] The VAD module 103a included in the bidirectional audio connector unit 103 identifies and analyzes speech segments received from the speech input of the user 101 in the user interaction session into speech segments and non-speech segments. The turn detection module 103b of the bidirectional audio connector unit 103 identifies and analyzes speech segments received for speech start, turn-over, and speech end in the user interaction session with the user 101. The bidirectional audio connector unit (103) determines and stores at least one of speech segments, non-speech segments, turn-over speech segments, and interruption speech segments in the user interaction session. The bidirectional audio connector unit 103 is further configured to identify and analyze speech segments received from the speech input of the user 101 in the user interaction session into speech segments and non-speech segments using a voice activity detector (hereinafter referred to as VAD) module 103a.

[0029] Furthermore, identification of speech segments, non-speech segments, speech onsets, speech ends, speaker turns, and user timeouts in received voice data or audio recordings of a user's speech input during an interaction session (such as a phone call with utterances) facilitates increased accuracy of audio data transcription, diary, speaker adaptation, and / or voice analysis. Furthermore, the speech turn detection module 103b is used to reduce user latency by detecting when the user's 101 speech input ends. The two-way audio connector unit 103 can further connect to a conversation controller module 109 to receive VAD and speech turn preference settings associated with the user's 101 profile once the user 101 is authenticated to the IVR communication system 100. The conversation controller module 109 is further configured to assign and modify thresholds for determining speech segments, non-speech segments, speech onsets, speech ends, speaker turns, and user timeouts.

[0030] The voice input of the user 101 is then routed to the speech / voice processing unit 104 of the IVR communication system 100. The speech / voice processing unit 104 corresponds to the ASR engine 104a. Those skilled in the art will recognize that the ASR engine 104a is capable of receiving and converting the voice input signal from the voice input of the user 101 into a text output. The ASR translation text represents an optimal analysis of the voice input signal of the words spoken in the voice input of the user 101 and the extra dialogue sounds using the corresponding ASR model that is optimal for the user 101. The conversation controller module 109 is further connected to the speech / voice processing unit 104. The conversation controller module 109 receives and analyzes the voice statistics derived from the ASR engine and assigns the optimal ASR model to the user 101. The conversation controller module 109 is further configured to update the ASR model preferences in the user's 101 profile information in the user profile database 110 for the optimization of the dialogue session. Furthermore, the conversation controller module 109 may also assign a corresponding ASR model to the user 101 for an interaction session without analyzing the speech statistics if there is no new and / or updated information corresponding to the ASR model available in the profile of the user 101. The conversation controller module 109 selects an ASR model for an interaction session assigned to the user 101 according to the user profile information available in the corresponding user profile database 110. The speech / voice processing unit 104 corresponds to a plurality of ASR models stored in an ASR model storage 104b that stores large vocabulary speech input recognitions for use during recognition in a user interaction session. The speech / voice processing unit 104 may further analyze the speech input of the user 101 using an emotion and sentiment recognition module 104c, track and analyze the user's emotional engagement during the user interaction session, and calculate corresponding emotion and sentiment scores, such as detecting the user's 101's emotion, mood, sentiment, etc.The speech / voice processing unit 104 may also analyze the audio data of the received speech input of the user 101 using a noise profile and environmental noise classification module 104d configured to distinguish the received voice from background noise. The speech processing unit (104) receives and analyzes the speech data and voice data features from the user's speech input and stores an ASR (Automatic Speech Recognition) model corresponding to the user interaction session.

[0031] In an exemplary, non-limiting scenario for one of the embodiments of the present invention, when the speaker's background noise and environmental noise level is detected to be higher than a threshold predetermined by the conversation controller module 109, the conversation controller module 109 may select a model dialogue engine 105 having a slow speaking rate and configured to repeat a portion of the dialogue engine to improve the user experience with the IVR communication system 100. Furthermore, rather than assigning a normal ASR, the conversation controller module 109 may select an alternative noise-robust ASR model for the user, such as a noise-robust ASR model that generally does not perform as well on clean audio data as compared to the normal ASR, but performs better than the normal ASR on noisy audio data.

[0032] Furthermore, if it is detected that the user 101 is struggling to provide long numeric inputs due to background noise, environmental noise, or speaking characteristics, the conversation controller module 109 modifies the corresponding dialogue engine model to design alternative conversation paths and break down the conversation for the user, such as, but not limited to, allowing the user to speak two digits at a time or five digits at a time. The conversation controller module 109 stores the modified dialogue engine model in the user profile database 110 as a user preference corresponding to the user's 101 profile.

[0033] Furthermore, the conversation controller module 109 is also configured to modify the corresponding dialogue engine model to adjust the speech rate according to the user preferences. During a dialogue session, if the user utterance of the voice input includes, for example, a sentence such as "please speak slowly" or "please speak fast", the controller module 109 modifies the speech rate of the model associated with the dialogue engine 105 accordingly. The conversation controller module 109 stores the preferred average speaking rate of the user 101 in the corresponding user profile in the user profile database 110. Furthermore, if the user utterance of the voice input includes, for example, a sentence such as "please repeat -", the conversation controller module 109 modifies the model associated with the dialogue engine 105 to repeat the relevant information, for example, at 0.9 times the speed. However, if it is determined that the user is not understood even after several repetitions, the utterance of the dialogue engine 105 is flagged for repronunciation.

[0034] Returning to FIG. 1, the audio signal received from the speech input of the user 101 is further analyzed using a voice biometrics module 104e. The voice biometrics module 104e corresponds to a voice biometrics system adapted to authenticate a user based on a voice diagnosis corresponding to the received voice audio input. The voice biometrics module 104e further connects to a user identification module 111 to perform user authentication. The user identification module authenticates the user to a user interaction session using the user's caller number or a unique identification number assigned to the user.

[0035] The voice biometrics module 104e matches one or more voiceprints corresponding to audio signals from the speech voice input of the user 101 received in an interaction session with one or more voiceprints of the user 101 stored from past interaction sessions. The voiceprints are stored in a user profile associated with the user 101 in the user profile database 110. The voice biometrics module 104e compares the one or more voiceprints for authentication of the user 101 to the IVR communication system 100.

[0036] The speech / voice processing unit 104 further connects to the dialogue engine 105. Those skilled in the art will recognize that the dialogue engine 105 is capable of receiving the transcribed text from the speech / voice processing unit 104 and performing corresponding logical processing. The dialogue engine 105 further drives the speech / voice processing unit 104 by providing a user interface between the user 101 and the service, primarily by conducting natural language dialogue with the user. The dialogue may have questions requesting one or more aspects of a particular service, such as asking for information. In this manner, the IVR communication system 100 may also receive general conversational queries and conduct ongoing conversations with the user via the dialogue engine 105. The dialogue engine 105 may further switch domains and use cases by recognizing new intents, use cases, contexts, and / or domains by the user during the conversation. The dialogue engine 105 holds and maintains the dynamic structure of the user interaction session as the interaction unfolds. A context, as used herein, is a collection of words, their meanings, and their relationships as understood in the current interaction of the user interaction session.

[0037] The dialogue engine 105 further includes an NLU component 105a. Those skilled in the art will recognize that the NLU component can receive input from the dialogue engine 105 and translate the natural language input into machine-readable information. The NLU component determines and generates transcribed context, intent, use case, entities, and metadata of the conversation with the user 101. The NLU component corresponds to a plurality of NLU models stored in the NLU model storage 105b, and determines a use case from the user's voice input as conversation data using natural language processing. The dialogue engine 105 further includes at least one of the NLU component 105a, the NLU model storage 105b, the dialogue engine core model database 105c, the action server 105d, and the dialogue state tracker 105e, which are disposed in the dialogue engine 105.

[0038] The dialogue state tracker module 105e is configured to track the "dialogue state", including, for example, providing hypotheses regarding the current state and / or analyzing the conviction state of the user's 101 goal or intent over the course of a user interaction session in real time. The dialogue state tracker 105e adds information related to the user interaction session. In determining the dialogue state, the dialogue state tracker module 105e determines the most inclined values ​​of corresponding slots and / or forms appropriate for the interaction based on the user's 101 speech input in the user interaction session and the corresponding conversation data model. The slots act as key-value stores used to store information provided by the user 101 during the interaction session and additional information collected, including, for example, the results of database queries. The types of slots can be in multiple forms, including, but not limited to, text slots, Boolean slots, category slots, float slots, etc. According to an embodiment of the present invention, the slot values ​​can affect the interaction session with the user 101 and affect the next action prediction. The dialogue engine 105 is further configured, using the action server 105d, to perform appropriate system actions and populate appropriate forms and / or slots corresponding to the user interaction session.

[0039] Returning to FIG. 1, the dialogue engine 105 further uses the dialogue engine core model storage 105c to determine the user's conversation actions in the user interaction session. The dialogue engine core model storage 105c further includes a prediction system for providing predicted conversation and response actions and conversation path design to the dialogue engine component and action server 105d. The predicted response, conversation actions and conversation path design include, for example and without limitation, generating a transcript for the spoken response, querying a corresponding database or making a phone call, generating at least one of a plurality of forms and / or slots, and validating one or more forms and / or slots in the user interaction session. The multiple forms and slots are used to better express the user's intent, and each slot in the corresponding conversation data model is filled through user interaction with the dialogue engine component and action server 105d. The dialogue engine core model storage 105c provides a flexible dialogue structure, allowing the user 101 to fill multiple slots in various orders in one user interaction session. The dialogue engine component and action server 105d associated with the corresponding dialogue engine core model running the application further uses the conversation data model to track the conversation context and history information, determine which slots have been filled with information from the user, and determine which slots need to be presented to complete the corresponding form.

[0040] In yet another embodiment of the present invention, slots are defined so as not to affect the flow of an interaction session with the user 101. In such a non-limiting exemplary scenario, the conversation controller module 109 receives and stores information related to the slots and / or forms in a user profile corresponding to the user 101 in the user profile database 110. The appropriate forms and / or slots are entered to add the individual's interaction history information to the interaction state tracker 105e.

[0041] Returning to FIG. 1 , the conversation controller module 109 further connects to the dialogue engine 105. The conversation controller module 109 receives and analyzes processed conversation data from the dialogue engine 105 corresponding to the user interaction session. The conversation controller module 109 then derives explicit user preferences for the NLU model, the dialogue engine, and the dialogue engine core model, and modifies the corresponding conversation data model of the user 101 within the user interaction session. The user profile of the user 101 in the user profile database 110 is then updated with the corresponding user preferences using the conversation controller module 109. Furthermore, the preferences are implemented in real-time during the interaction session with the user and are also stored for optimization of future user interaction sessions with the user 101 by the conversation controller module 109.

[0042] The interaction state tracker module 105e is further connected to the session monitoring module 108. The session monitoring module 108 is further configured to determine a session state of the user interaction session and record session state related information between the user 101 and the service instance, such as username, client, timestamp, session state, etc., where the session state is logged in, active, disconnected, connected, logged off, etc. The session monitoring module 108 is further configured to add a session ID of the corresponding user interaction session, add user metrics, and explicitly and automatically calculate a happiness index of the user 101 during the interaction session. The happiness index is calculated by applying a weight to each type of information based on information received during the user interaction session with the user 101.

[0043] The session monitoring module 108 further connects to the conversation controller module 109. The conversation controller module 109 is further configured to receive conversation statistics derived by the session monitoring module 108, such as, but not limited to, a happiness index score, user session metrics, and aggregated user session metrics corresponding to a session ID associated with the user 101 for optimizing the user journey of the user 101. The conversation controller module 109 further updates a user profile of the user 101 in the user profile database 110 with the corresponding happiness index score, user session metrics, and aggregated user session metrics. The session monitoring module monitors the user interaction session and adds key metrics corresponding to the user interaction session to the conversation controller module 109. The added key metrics include at least one of a confidence score, a level of expertise of the user, a number of application forms / slots, a conversation length, a fallback rate, a retention rate, and a goal achievement rate.

[0044] The conversation controller module 109 is further configured to make necessary modifications based on the received conversation statistics to optimize the user interaction session. The conversation controller 109 is also configured to assign and modify thresholds for determining non-speech segments.

[0045] The conversation controller 109 is further configured to select and / or modify a conversation data model based on the received audio characteristics and / or an existing user profile.

[0046] In a non-limiting exemplary scenario for one of the embodiments of the present invention, scores corresponding to emotions and sentiments detected from the speech input of the user 101 and the engagement of the user 101 are used to infer an average satisfaction level of the user 101. The conversation controller module 109 can compare the estimated satisfaction level with a happiness index score provided by the user 101. The conversation controller module 109 then aims to modify corresponding parameters, such as models associated with the dialogue engine 105, to optimize the user interaction session. The dialogue engine 105 is further configured to predict an appropriate system action based on the received conversation data using the dialogue engine core model store 105c, where the appropriate system action can be: Generate a transcript of the spoken response - Querying the corresponding database or making a call Generate at least one of a plurality of forms and / or slots. Validating one or more forms and / or slots Contains at least one of the following:

[0047] Appropriate system actions for the conversation data are determined using at least one of a Transformer Embedded Interaction (TED) policy, a storage policy, and a rules policy.

[0048] Moreover, in another non-limiting exemplary scenario for one of the embodiments of the present invention, an alternative conversation path is uniquely selected by the conversation controller module 109 based on the user expertise levels of multiple users and corresponding past session metrics. Multiple conversation path options are tested for the user by the conversation controller module 109, and based on the corresponding session metrics, a preferred conversation path is stored in the user profile of the associated user in the user profile database 110. This is performed by inferring a preferred conversation path option based on a session in which the conversation controller module 109 instructed the dialogue engine to test various options of conversation paths for the user.

[0049] Further, in another non-limiting exemplary scenario, a feedback classifier model is included in the happiness index feedback block with the user's reaction compared to the experience. A number of different dependent parameters are adjusted based on the response category by the conversation controller module 109. The conversation controller module 109 further stores the corresponding parameters of the corresponding user profile associated with the user in the user profile database 110 for future optimization.

[0050] Furthermore, in another non-limiting exemplary scenario, when modifying a model associated with the dialogue engine based on a user's expertise level, after multiple successful user dialogue sessions, the dialogue controller module 109 increases the speaking rate. If it is determined that the user 101 does not face challenges at the increased speaking rate, the dialogue controller module 109 stores the speaking rate as a corresponding preferred rate associated with the user profile. Otherwise, the original speaking rate will be reverted or slowed down.

[0051] Returning to Figure 1, the dialogue state tracker module 105e further connects to the dialogue engine dispatcher 106. The dialogue engine dispatcher 106 is configured to receive the input response actions from the dialogue state tracker module 105e, generate corresponding responses in the form of one or more text message information, and deliver them to the TTS module 107. The dialogue engine dispatcher 106 can further store multiple responses in a queue until they are delivered to the TTS module 107. The dialogue engine dispatcher generates responses corresponding to the user's intent during the user interaction session.

[0052] The TTS module 107 receives the response and recognizes the text message information. The TTS module 107 applies corresponding TTS models and corresponding TTS parameters from the TTS model storage 107a and the TTS parameter database 107b, respectively, to perform voice synthesis and generate a voice narration. The TTS model storage 107a stores a plurality of TTS models including phonemes of text and voice data corresponding to phonemes of natural language voice data and / or synthesized voice data. The TTS parameter database 107b stores parameters related to voice attributes of the voice response output, including, but not limited to, language type, voice gender, voice age, voice speed, volume, tone, pronunciation of special words, breaks, accent, intonation, etc. As a result, the TTS module 107 converts the received response from the dialogue engine dispatcher 106 into a voice speech response for the user 101. The TTS module (107) receives the generated response and performs voice synthesis. The process of performing speech synthesis on the generated responses further comprises storing a TTS model and TTS parameters such as speaking rate, pitch, volume, intonation, preferred responses, etc. corresponding to the user interaction session.

[0053] The TTS module 107 is further connected to a dialogue controller module 109. The dialogue controller module 109 is further configured to select an assigned TTS model for a user interaction session according to user preferences set forth in a corresponding user profile of the user 101 in the user profile database 110. The dialogue controller module 109 is further configured to modify and update preferences of the corresponding TTS model in the user profile of the user 101 in the user profile database 110 for optimization of the interaction session.

[0054] In a non-limiting exemplary scenario for one embodiment of the present invention, the dialogue controller module 109 selects an appropriate TTS model based on the emotion score of the user 101. The TTS model corresponds to a different voice, such that if the user 101 sounds angry, the dialogue controller module 109 selects a TTS model that corresponds to an empathetic voice.

[0055] In another exemplary embodiment of the invention, a large-scale language model module 125a (hereinafter, "LLM module") is included in the dialogue engine 105. FIG. 1B is a conceptual diagram illustrating an exemplary framework for monitoring and optimizing a user dialogue session in such an embodiment with respect to the overall data transmitted and received in a dialogue session between a human user and an exemplary interactive voice response (IVR) communication system 120. In this example, the system disclosed herein includes the LLM module 125a in the IVR communication system 120 of FIG. 1B. For convenience of explanation, in this exemplary embodiment, the LLM module 125a is a large-scale language model for determining a use case from a user's voice input as dialogue data and generating a reply text based on the received dialogue data information. Furthermore, to make the execution of a task, including generating a reply, flexible, a set of rules based on natural language prompts, task descriptions, and formatting instructions are introduced into the LLM module 125a to be followed. In this exemplary embodiment, the LLM language model can help the IVR communication system 120 to more efficiently interpret the natural language input of the user 101 and provide support for continuous optimization and updating of the IVR communication system 120. In this exemplary embodiment, the dialogue engine 105 can receive the transcribed text from the speech / voice processing unit 104 and perform corresponding logical processing using the LLM module 125a. The dialogue engine 105 includes at least one of a large-scale language module 125s, an action server 105d, and a dialogue state tracker 105e disposed within the dialogue engine 105.

[0056] The dialogue engine 105 further drives the speech / voice processing unit 104 by conducting natural language dialogue with the user using the LLM module 125a, primarily providing a user interface between the user 101 and the service. The dialogue may have questions requesting one or more aspects of a particular service, such as asking for information to more precisely understand the intent of the user 101. In this way, the IVR communication system 120 can also receive general conversational queries and engage in a continuous dialogue with the user via the dialogue engine 105. Using the LLM module 125a, the dialogue engine 105 can further generate coherent sentences, consider user inputs, generate responses along a conversation trajectory, participate in multi-turn conversations, and handle open-ended inquiries. Furthermore, when the dialogue engine 105 has to provide translation of user input or summarize long responses, and switch domains and use cases by recognizing new intents, use cases, contexts, and / or domains by the user during the conversation, the LLM module 125a helps to guide the dialogue session in the direction of the user's intent, and the action server 105d generates prompts that provide the LLM module 125a with the necessary context and information to generate personalized, coherent, and relevant responses. Furthermore, after receiving the generated text response from the LLM module 125a, the action server 105d can perform post-processing to refine the generated text. Post-processing includes removing redundant information, formatting the text, and adding context.

[0057] Along with slot tracking, the action server 105d is further capable of interfacing with external APIs, sending requests, receiving responses, and integrating information into a conversation within an interactive session.

[0058] Additionally, using the LLM module 125a and the action server 105d, the dialogue engine 105 retains and maintains the dynamic structure of a user dialogue session as the dialogue unfolds, making the dialogue feel more like a human conversation. Context, as used herein, refers to the collection of words and their meanings and relationships that are understood within the current dialogue of a user dialogue session.

[0059] The dialogue state tracker module 105e is configured to track the "dialogue state", including, for example, providing hypotheses regarding the current state and / or analyzing the confidence state of the user's 101 goals or intents over the course of a user dialogue session in real time. In determining the dialogue state, the dialogue state tracker module 105e determines the most inclined values ​​of the appropriate corresponding slots and / or forms in the dialogue based on the user's 101 speech input in the user dialogue session and the corresponding dialogue data model. Furthermore, in cooperation with the action server 105d and the LLM module 125a in the dialogue engine 105, the dialogue state tracker 105e coordinates the dialogue to ensure a coherent and contextually relevant dialogue. The dialogue state tracker 105e works closely with the action server 105d to identify which slots are required to complete an intent or task, and tracks the context of the dialogue, including the user's previous input, the system's responses, and any relevant information exchanged. This helps ensure that the dialogue in a dialogue session remains coherent and relevant across multiple turns. During a user interaction session with the user 101, the interaction state tracker 105e updates the interaction state with new information. For example, when the user provides a response that fills a slot in a form, the interaction state tracker 105e records this information. The interaction state tracker 105e helps determine when to invoke the action server 105d to gather missing information and when to involve the LLM module 125a to generate a response. The interaction state tracker 105e provides the necessary information to both the LLM module 125a and the action server 105d to ensure a seamless interaction session.

[0060] FIG. 2 is a flow chart 200 illustrating an example speech and / or non-speech segment detection process flow in an example scenario in the context of the IVR communication system 100 shown in FIG.

[0061] In step 201, a user interaction session is initiated with the user 101, and the dialogue controller module 109 determines, selects or modifies a VAD model and a turn-taking model assigned to the user 101 corresponding to the user profile in the user profile database 110. The dialogue controller module 109 further determines and selects a final non-voice segment threshold for determining non-voice segments of a received speech signal corresponding to a speech input of the user 101. The non-voice segments are detected based on a comparison with a preset final non-voice segment detection threshold.

[0062] In the next step, in step 202, an audio signal corresponding to a user voice input from the user 101 is received via the telecommunications channel 102. The bidirectional audio connector unit 103 further analyses the audio signal during the user interaction session.

[0063] In the next step, in step 203, the corresponding VAD model and turn-taking model determined and selected in step 201 are used to detect and identify speech segments, non-speech segments, speech starts, speech ends, turn-taking and user timeouts in real-time within the received user dialogue audio data.

[0064] In a non-limiting exemplary scenario of one of the embodiments of the invention, the decision-making core of speech and / or non-speech segment detection is represented by a VAD module 103a combined with a sliding window approach. For example, each input audio frame of length, for example 20 milliseconds, arriving at the bidirectional audio connector unit 103 is classified by a VAD tool. The VAD module 103a can currently be implemented using multiple interfaces such as Google's WebRTCVAD. The VAD tool 103a further determines whether a speech frame is generated or not. The audio frame and the classification result are inserted into a ring buffer (not shown here for simplicity) configured with a padding period specified, for example, in milliseconds. The ring buffer is implemented in a sliding window VAD class and is responsible for implementing at least one of the interfaces "CheckStart(AudioFrame frame)" and "checkEnd(AudioFrame frame)".

[0065] The characteristics corresponding to the voiced and non-voiced segments are further defined and stored in a database (not shown here for simplicity) that comprises a number of models associated with speech and non-speech segments.

[0066] For example, for "start of speech" segment detection, it is detected that user 101 has started speaking when the number of incoming voice frames with speech in the ring buffer is greater than the activation threshold times the maximum buffer size. In such a case, all incoming voice frames in the ring buffer are sent to the ASR engine, as well as all subsequent incoming voice frames, until the end of user 101's speech is detected.

[0067] Additionally, if an end-of-speech segment is detected, a new frame is inserted into the ring buffer via the "checkEnd" method when the user is speaking. After inserting the new frame, the speaker change module 103b determines whether the number of non-speech frames is greater than the deactivation threshold times the maximum buffer size. In such a case, the speaker change module 103b determines the "USER_HAS_STOPPED_SPEAKING" state.

[0068] Furthermore, in case of final non-speech segment detection, during the speaker change corresponding to the USER_HAS_STOPPED_SPEAKING state, the VAD module 103a determines whether the final non-speech threshold has been reached or whether the user 101 has started speaking again by comparing the current time with the timestamp of the last speech end event. Start of speech detection works as described in paragraph 0054 above. If the VAD module 103a determines that the final non-speech segment threshold has been reached, the ASR engine is notified and then waits for the transcription result. It then determines that the turn of the current user 101 is complete.

[0069] For example, if the user 101 stops speaking, the user 101 needs to be detected before the final non-speech segment threshold is reached, e.g., given a default configuration and a final non-speech segment threshold of 1500 ms, the user 101 needs to start speaking again after about 1200 ms so that speech input is detected before the final non-speech segment threshold is reached. This is because the ring buffer needs to contain a maximum ring buffer size frame of times the activation threshold with speech, which may be, e.g., 300 ms, so that the user 101 speech input is detected and also includes some margin for classification errors / true negatives. Furthermore, in case of user timeout detection, a timeout threshold of, e.g., 5 seconds, is also set by default when the IVR communication system 100 is waiting for the user 101 to start speaking. If speech start is not detected within 5 seconds, the user timeout is detected by the VAD module 103a and the turn of the user 101 is ended.

[0070] In a final step 204, the audio frames corresponding to the speech segments are stored in the two-way audio connector unit 103. The process ends in step 204 and the audio frames are then sent further to the ASR engine 104a in the speech / audio processing unit 104 for transcription.

[0071] FIG. 3 is a flow chart 300 illustrating the flow of conversation controller integration in an exemplary scenario in the context of the IVR communication system 100 shown in FIG.

[0072] In step 301, a user, such as user 101, initiates a call to the IVR communication system 100, for example through a telephony gateway. The call is transmitted to the IVR communication system 100 over a telecommunications channel 102.

[0073] In the next step, an interaction session is established in step 302. The two-way audio connector unit 103 receives and analyzes audio signals corresponding to user speech input from the user 101 during the user interaction session. After user identification by the user's caller number or unique ID, the two-way audio connector unit 103 stores and identifies speech segments, non-speech segments, speech starts, speech stops, speaker turns, and user timeouts in real time in the received audio data of the user interaction. The voice segments are then sent to the speech / voice processing unit 104.

[0074] In the next step, in step 303, the speech segment is received by the speech / voice processing unit 104 and features of the received audio data of the speech segment are suitably analysed for audio statistics, e.g. emotion, sentiment, noise profile, environmental audio information etc.

[0075] In the next step, in step 304, the conversation controller module 109 then receives and analyzes the speech statistics derived from the speech segments and assigns and / or modifies an ASR model that best suits the user interaction session corresponding to the user's 101 profile.

[0076] In the next step, in step 305, the ASR engine transcribes the speech segment into machine-readable text that corresponds to the ASR model assigned by the conversation controller module 109. The transcribed machine-readable text is then sent to the dialogue engine 105.

[0077] In a next step, the dialogue engine 105 receives the transcribed machine-readable text, step 306. The conversation controller module 109 further assigns and / or modifies NLU models based on the derived speech statistics received from the speech segments in step 304 to best suit the user dialogue session and correspond to the profile of the user 101 in the user profile database 110.

[0078] In the next step, in step 307, the NLU component classifies and understands the domain and the user's intent in the user's utterance. The NLU component further extracts entities and classifies the roles of the entities by performing syntactic analysis and semantic analysis. In syntactic analysis, the user's utterance is separated into syntactic units (words, phases, morphemes, etc.) and the syntactic elements in the separated units are understood. The semantic analysis is performed using at least one of semantic matching, rule matching, and formula matching. The NLU component further performs classification of intent, domain, entity, entity role, and role corresponding to the NLU model assigned by the conversation controller module 109 in step 304.

[0079] In a next step, at step 308, the dialogue state tracker module 105e tracks the "dialogue state" including, for example and without limitation, providing hypotheses regarding the current state and / or analyzing the belief state of the user's 101 goals or intents in real time over the course of the user dialogue session. The dialogue state tracker module 105e further adds the latest "dialogue state" to the conversation data model.

[0080] In the next step, in step 309, the dialogue controller module 109 further assigns and / or modifies the dialogue data model based on the dialogue engine core model stored in the dialogue engine core model storage 105c and / or derived speech statistics received from the speech segments to best suit the user dialogue session and to correspond to the user's profile 101 in the user profile database 110. The dialogue engine core model storage 105c provides a flexible dialogue structure, allowing the user 101 to fill multiple slots in various orders during one user dialogue session. The dialogue engine core model assigned by the dialogue controller module 109 further predicts appropriate actions for the dialogue story using one or more machine learning policies, such as, but not limited to, a Transformer Embedding Dialogue (TED) policy, a storage policy, and a rule policy. The Transformer Embedding Dialogue (TED) policy is a multi-task architecture for next action prediction and entity recognition. This architecture consists of multiple transformer encoders shared for both tasks. The storage policy stores the stories from the training data. The memorization policy involves checking if the current conversation matches a story in the corresponding story file. In that case, the memorization policy is useful for predicting the next action from the matching story in the corresponding training data. The rule policy is a policy that processes the part of the conversation that follows a certain behavior (such as business logic). It makes predictions based on any rules contained in the corresponding training data.

[0081] In the next step, in step 310, the dialog engine component and action server module 105d executes the action predicted by the dialog engine core model assigned by the conversation controller module 109. The dialog engine component and action server module 105d executes custom actions such as, but not limited to, API calls, database queries, adding an event to a calendar, checking the user's 101 bank balance, etc.

[0082] In the next step, in step 311, the dialogue state tracker module 105e sends the latest "dialogue state" to the conversation controller module 109 and the session monitoring module 108. The conversation controller module 109 receives the latest "dialogue state" and accordingly updates a corresponding number of models and / or user preferences associated with the user profile of the user 101 in the user profile database 110 with the conversation statistics.

[0083] In the next step, in step 312, the session monitoring module 108 extracts and adds conversation statistics, such as, but not limited to, a session ID of the corresponding user interaction session, adds user metrics, and calculates explicit and automatic happiness indices of the user 101 in the user interaction session. The session monitoring module 108 extracts and adds the conversation statistics to the conversation controller module 109. The conversation controller module updates the associated user profile accordingly. It should be understood that steps 311 and 312 are performed simultaneously according to an embodiment of the present invention.

[0084] Furthermore, after step 310, in step 313, which is executed in parallel with step 311, the dialogue engine dispatcher 106 receives the input of the response action and generates and delivers a corresponding response in the form of one or more text message information. The dialogue engine dispatcher 106 further stores the multiple responses in a queue until they are delivered accordingly.

[0085] In the next step, in step 314, the response is sent to the TTS module 107. The TTS module 107 receives the response and recognizes the text message information. The TTS module 107 applies the corresponding TTS model and corresponding TTS parameters from the TTS model storage 107a and the TTS parameter database 107b, respectively, to perform speech synthesis and generate a voice narration. The dialogue controller module 109 further assigns and / or modifies the TTS model and corresponding TTS parameters based on the derived voice statistics received from the speech segments to best suit the user interaction session and correspond to the profile of the user 101 in the user profile database 110.

[0086] It should be appreciated that steps 312 and 314 are performed simultaneously according to one embodiment of the present invention.

[0087] It will be appreciated that a user interaction management method for monitoring and optimizing a user interaction session includes the steps of determining speech segments, non-speech segments, turn-taking speech segments, and interruption speech segments; receiving and analyzing conversation data and voice features from a user's speech input; receiving the voice features and selecting and / or modifying associated ASR (automatic speech recognition) models and NLU (natural language understanding) models for the user interaction session; receiving and processing transcribed text corresponding to the conversation data; adding information related to the user interaction session; monitoring the user interaction session and adding key metrics; generating responses corresponding to the user's intent during the user interaction session and performing speech synthesis on the generated responses.

[0088] The steps of determining the speech segments, non-speech segments, turn-taking speech segments, and barge-in speech segments further include receiving an assigned model for determining the speech segments, receiving an assigned threshold for determining the non-speech segments, listening to the user's speech input, applying the assigned model for determining the speech segments, applying the assigned threshold for detecting the non-speech segments, and storing and transmitting the speech input for speech processing.

[0089] Additionally, performing speech synthesis on the generated response also includes identifying and parsing speech segments received from the speech input of the user 101 in the user interaction session into speech and non-speech segments. The process further includes adjusting a speaking rate of the generated response corresponding to the received speech characteristics and / or an existing user profile.

[0090] Adding information related to the user interaction session further includes updating and training ASR and NLU models associated with the registered user profile using audio data of the collection of user speech audio from the corresponding user interaction session, and updating and training models associated with determining speech segments, non-speech segments, turn-taking speech segments, and interruption speech segments associated with the registered user profile using features of the audio data of the collection of user speech audio from the corresponding user interaction session. Adding dialogue information related to the user interaction session further includes performing determined system actions and filling in appropriate forms and / or slots.

[0091] Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the preceding description, numerous specific details, such as various example configurations, are provided to provide a thorough understanding of embodiments of the described technology. However, one of ordinary skill in the relevant art will recognize that the technology can be practiced without one or more of the specific details, or with other methods, components, devices, etc. In other instances, well-known structures or operations have not been shown or described in detail to avoid obscuring aspects of the technology.

[0092] Although the subject matter has been described in language specific to structural features and / or operations, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features and operations described above. Rather, the specific features and acts described above are disclosed as example forms for implementing the claims. Numerous modifications and alternative arrangements can be devised without departing from the spirit and scope of the described technology.

[0093] Furthermore, the described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments. In the preceding description, numerous specific details, such as various example configurations, are provided to provide a thorough understanding of embodiments of the described technology. However, one of ordinary skill in the relevant art will recognize that the technology can be practiced without one or more of the specific details, or with other methods, components, devices, etc. In other instances, well-known structures or operations have not been shown or described in detail to avoid obscuring aspects of the technology.

[0094] Although the subject matter has been described in language specific to structural features and / or operations, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features and operations described above. Rather, the specific features and acts described above are disclosed as example forms for implementing the claims. Numerous modifications and alternative arrangements can be devised without departing from the spirit and scope of the described technology.

Claims

1. A user interaction management system (100) for monitoring and optimizing a user interaction session in an interactive voice response system with human-computer interaction, comprising: a bidirectional audio connector unit (103) for determining and storing at least one of a speech segment, a non-speech segment, a turn-taking speech segment, and an interruption speech segment in a user interaction session; a user identification module (100) for authenticating the user to a user interaction session using the user's caller number or a unique identification number assigned to the user; a speech / voice processing unit (104) for receiving and analyzing speech data and voice data features from a user speech input and storing an ASR (Automatic Speech Recognition) model corresponding to said user interaction session; a dialogue engine (105) for receiving and processing transcribed text corresponding to the conversation data, storing corresponding dialogue engine components and natural language understanding (NLU) models, and for processing voice-based dialogue with the user during the user dialogue session; an interaction state tracker (105e) for adding information related to said user interaction session; a conversation controller module (109) for receiving the speech features and selecting and / or modifying associated ASR and NLU models for the user conversation session to optimize a duration of the user conversation session; a session monitoring module (108) for monitoring the user interaction session and adding key metrics corresponding to the user interaction session to the conversation controller module (109); a dialogue engine dispatcher (106) for generating responses corresponding to the intent of the user during the user interaction session; a Text-to-Speech (TTS) module (107) for receiving the generated response and for performing speech synthesis; Equipped with The user interaction management system (100) for monitoring and optimizing user interaction sessions.

2. the conversation controller (109) is further configured to select and modify models related to determining speech segments, non-speech segments, turn-taking speech segments, and interruption speech segments to achieve a target service. A user interaction management system (100) for monitoring and optimizing a user interaction session in an interactive voice response system during human-computer interaction according to claim 1.

3. The dialogue controller (109) is further configured to assign and modify thresholds for determining non-speech segments. A user interaction management system (100) for monitoring and optimizing a user interaction session in an interactive voice response system during human-computer interaction according to claim 1.

4. The conversation controller (109) is further configured to select and modify a conversation data model based on the received speech characteristics and / or an existing user profile. A user interaction management system (100) for monitoring and optimizing a user interaction session in an interactive voice response system during human-computer interaction according to claim 1.

5. The dialogue engine 105 further comprises at least one of the NLU component 105a, an NLU model storage 105b, a dialogue engine core model database 105c, an action server 105d, and the dialogue state tracker 105e disposed within the dialogue engine 105; A user interaction management system (100) for monitoring and optimizing a user interaction session in an interactive voice response system during human-computer interaction according to claim 1.

6. The dialogue engine 105 further comprises at least one of a large-scale language module 125s, an action server 105d, and the dialogue state tracker 105e disposed within the dialogue engine 105; A user interaction management system (100) for monitoring and optimizing a user interaction session in an interactive voice response system during human-computer interaction according to claim 1.

7. The dialogue engine (105) is further configured to predict, using a dialogue engine core model storage (105c), an appropriate system action based on the received conversation data, where the appropriate system action comprises: generating a transcription of the spoken response; Querying a corresponding database or making a call; generating at least one of a plurality of forms and / or slots; validating said plurality of forms and / or slots; Including, A user interaction management system (100) for monitoring and optimizing a user interaction session in an interactive voice response system during human-computer interaction according to claim 1.

8. the appropriate system action for the conversation data is determined using at least one of a Transformer Embedded Dialogue (TED) policy, a storage policy, and a rules policy. A user interaction management system (100) for monitoring and optimizing a user interaction session in an interactive voice response system during human-computer interaction according to claim 7.

9. The speech processing unit (104) is further configured to detect and analyze at least one of emotion, sentiment, noise profile, and environmental audio information from the received audio data features. A user interaction management system (100) for monitoring and optimizing a user interaction session in an interactive voice response system during human-computer interaction according to claim 1.

10. the dialogue engine (105) is further configured to use an action server (105d) to execute the appropriate actions and populate the appropriate forms and / or slots corresponding to the user dialogue session. A user interaction management system (100) for monitoring and optimizing a user interaction session in an interactive voice response system during human-computer interaction according to claim 7.

11. the appropriate forms and / or slots are entered to add the individual's interaction history information to the interaction state tracker (105e); A user interaction management system (100) for monitoring and optimizing a user interaction session in an interactive voice response system during human-computer interaction according to claim 10.

12. The user identification module (111) is further configured to distinguish between a synthesized voice in the received conversation data and a human voice of the user from the stored voice biometrics of the user in order to detect any fraudulent activity. A user interaction management system (100) for monitoring and optimizing a user interaction session in an interactive voice response system during human-computer interaction according to claim 1.

13. the user identification module (111) is further configured to store and update a user profile together with past and current dialogue session states and call session statistics, an ASR model, a dialogue engine model, and a TTS model corresponding to the existing user profile; A user interaction management system (100) for monitoring and optimizing a user interaction session in an interactive voice response system during human-computer interaction according to claim 1.

14. said speech / voice processing unit (104) is further adapted to use a voice biometrics module (104e) to identify and register said users participating in said user interaction session using voice biometrics. A user interaction management system (100) for monitoring and optimizing a user interaction session in an interactive voice response system during human-computer interaction according to claim 1.

15. The session monitoring tool (108) is further configured to determine and calculate a happiness index of the user in real time during the user interaction session. A user interaction management system (100) for monitoring and optimizing a user interaction session in an interactive voice response system during human-computer interaction according to claim 1.

16. said two-way audio connector unit (103) is further adapted to identify and analyse said audio segments received from the speech input of said user 101 during said user interaction session into speech segments and non-speech segments using a voice activity detector (hereinafter VAD) module (103a); A user interaction management system (100) for monitoring and optimizing a user interaction session in an interactive voice response system during human-computer interaction according to claim 1.

17. 1. A user interaction management method for monitoring and optimizing a user interaction session in an interactive voice response system during human-computer interaction, comprising: determining speech segments, non-speech segments, turn-taking speech segments, and interruption speech segments; receiving and analyzing speech data and audio features from a user speech input; receiving the speech features and selecting and / or modifying associated ASR (automatic speech recognition) and NLU (natural language understanding) models; receiving and processing a transcribed text corresponding to the conversation data; adding information related to said user interaction session; monitoring said user interaction sessions to add key metrics; generating a response associated with an interaction of the user during the user interaction session; performing speech synthesis on the generated response; Including, A user interaction management method for monitoring and optimizing a user interaction session.

18. The step of determining the speech segments, non-speech segments, turn-taking speech segments, and interruption speech segments further includes: receiving a model assigned to determine a speech segment; receiving a threshold assigned to determine a non-speech segment; listening to a user speech input; applying the assigned model to determine the speech segments; applying an assigned threshold value to determine the non-speech segments; storing and transmitting the speech input audio for speech processing; Including, 20. A user interaction management method for monitoring and optimizing a user interaction session in an interactive voice response system during human-computer interaction according to claim 17.

19. The step of processing speech synthesis on the generated response further comprises: identifying and parsing the audio segments received from the speech input of the user 101 during the user interaction session into speech segments and non-speech segments; Including, 20. A user interaction management method for monitoring and optimizing a user interaction session in an interactive voice response system during human-computer interaction according to claim 17.

20. The step of performing speech synthesis on the generated response further comprises: adjusting a speaking rate for the generated response corresponding to the received voice characteristics and / or an existing user profile. Including, 20. A user interaction management method for monitoring and optimizing a user interaction session in an interactive voice response system during human-computer interaction according to claim 17.

21. The step of adding information related to the user interaction further comprises: updating and training the ASR model and the NLU model associated with an enrolled user profile using the audio data of the collection of user speech utterances from the corresponding user interaction session; updating and training the models associated with determining speech segments, non-speech segments, turn-taking speech segments, and interruption speech segments associated with an enrolled user profile using the audio data features of the collection of user speech utterances from the corresponding user interaction session; Including, 20. A user interaction management method for monitoring and optimizing a user interaction session in an interactive voice response system during human-computer interaction according to claim 17.

22. The steps of receiving and processing transcribed text corresponding to the conversation data and processing a voice-based interaction with the user during the user interaction session further include: predicting appropriate system actions based on said received conversation data. wherein the appropriate system action comprises: generating a transcription of the spoken response; Querying a corresponding database or making a call; generating at least a plurality of forms and / or slots; Validating said one or more forms and / or slots. At least one of 20. A user interaction management method for monitoring and optimizing a user interaction session in an interactive voice response system during human-computer interaction according to claim 17.

23. The step of adding interaction information related to the user interaction session further comprises: performing the determined system action to populate the appropriate forms and / or slots. Including, 20. A user interaction management method for monitoring and optimizing a user interaction session in an interactive voice response system during human-computer interaction according to claim 17.

24. The step of receiving and analyzing speech data and voice features from the user speech input further includes: authenticating the user to a user interaction session using the user's caller number or a unique identification number assigned to the user; Including, 20. A user interaction management method for monitoring and optimizing a user interaction session in an interactive voice response system during human-computer interaction according to claim 17.

25. The step of authenticating the user to the user interaction session using the user's caller number or a unique identification number assigned to the user further comprises: utilizing voice biometrics to identify and register said users to participate in said user interaction session. Including, 25. A user interaction management method for monitoring and optimizing a user interaction session in an interactive voice response system during human-computer interaction according to claim 24.

26. predicting an appropriate system action based on the received conversation data using at least one of a Transformer Embedded Dialogue (TED) policy, a storage policy, and a rules policy.

23. A user interaction management method for monitoring and optimizing a user interaction session in an interactive voice response system during human-computer interaction according to claim 22.

27. The added key metrics include at least one of the following: user expertise level, number of application forms and / or slots, fallback rate, retention rate, and goal achievement rate; 20. A user interaction management method for monitoring and optimizing a user interaction session in an interactive voice response system during human-computer interaction according to claim 17.

28. said appropriate forms and / or slots are populated with service-oriented information received from said user to achieve said goals and / or targeted services; 22. A user interaction management method for monitoring and optimizing a user interaction session in an interactive voice response system during human-computer interaction according to claim 21.

29. The step of performing speech synthesis on the generated response further comprises: storing a TTS model and TTS parameters, such as speaking rate, pitch, volume, intonation, and preferred responses, corresponding to said user interaction session; Including, 20. A user interaction management method for monitoring and optimizing a user interaction session in an interactive voice response system during human-computer interaction according to claim 17.

Images