Adaptive speech recognition system with user feedback analysis and dynamic retriggering

Abstract

A computing system providing an enhanced user experience with a voice assistant (VA) that may be part of an automotive or vehicle infotainment system. A speech recognition module 10 converts spoken in

Description

BACKGROUND In the realm of automotive infotainment systems, Voice Assistants (VA) have gained significant traction, leveraging a blend of natural language processing (NLP), speech recognition, and machine learning technologies. These advanced systems are designed to interpret human speech, convert it into text, and process the text to perform specific tasks or commands. The goal is to provide a seamless, hands-free experience for users, particularly in the context of driving where manual interaction can be a safety concern. However, the practical implementation of such systems can be challenging due to a variety of factors. For instance, the input commands provided to voice assistants may not always yield the intended response. This could be due to a myriad of reasons such as the command not being received in the correct order or issues related to dialect and pronunciation. These factors can prevent the speech recognition engine from accurately decoding the input command. This issue becomes even more pronounced in a dynamic and often noisy environment like a vehicle. Furthermore, the effectiveness of the voice assistant's response can be subjective and dependent on the user's context, expectations, and preferences. For instance, a response that might seem satisfactory in one context could be perceived as inadequate or incorrect in another. This subjectivity adds another layer of complexity to the design and implementation of such systems. On the other hand, alternative methods to improve the accuracy of voice assistants have been explored. For instance, incorporating visual cues or gestures as additional input modalities can potentially enhance the system's understanding of the user's intent. However, these methods come with their own set of challenges such as the need for additional hardware or the difficulty in accurately interpreting non-verbal cues. Therefore, there is a need to overcome the problems discussed above. A solution that can accurately interpret and process voice commands, handle varying dialects and pronunciations, and adapt to the user's context and preferences would be highly beneficial. Furthermore, the solution should be robust enough to function effectively in a dynamic environment such as a vehicle. In addition, it would be advantageous if the solution could learn and adapt from past interactions, thereby improving its performance over time. SUMMARY The primary objective of the present invention is to provide an enhanced user experience with a voice assistant (VA) by incorporating facial expression, emotional feedback, and gesture recognition. This allows the VA to detect incorrect or unexpected responses based on the user's non-verbal feedback, and reinitiate the session by asking the user to repeat their command. Another objective of the present invention is to utilize Deep Neural Network (DNN) and Convolutional Neural Network (CNN) models for recognizing facial expressions, emotions, and gestures. The output from these models is used to inform the VA's response, improving the accuracy and relevance of its interactions with the user. Yet another objective of the present invention is to leverage hardware such as RGB cameras, depth sensors like Microsoft Kinect or LiDAR sensors for accurate facial feature tracking and expression analysis. This hardware can also be used for gesture recognition by tracking the movement of hands or body parts. Still another objective of the present invention is to use machine learning and computer vision techniques to process the data from these sensors. Libraries and frameworks like OpenCV, TensorFlow, and PyTorch can be used to develop applications for recognizing facial expressions and gestures. According to one aspect of the present invention, a computing system comprises a speech recognition module 10, a processing unit 12, a feedback detection module 14, and a control module 18. The speech recognition module 10 is configured to receive an input command in the form of speech and convert it into text. The processing unit 12 transmits the converted text to a backend system 16 for further processing. According to another aspect of the present invention, the feedback detection module 14 designed to analyze user feedback associated with the response to the input command. It detects indicators of an incorrect response, which include facial expressions, gestures, or emotional states. When an indicator of an incorrect response is detected, the control module reinitiates the session by prompting the user to repeat the input command. In another embodiment of the present invention, the feedback detection module 14 further comprises a deep neural network (DNN) trained to classify user facial expressions as positive, neutral, or negative. This module can also use a convolutional neural network (CNN) to process real-time video data of the user’s facial expressions. In yet another embodiment, the control module 18 can initiate a different response based on the detected emotional state, such as providing suggestions to clarify the input command. The gesture input can be detected using a sensor interface comprising touch-based or non-touch-based input sensors. In a further embodiment, the feedback detection module 14 is configured to detect changes in user feedback over time and update the retriggering threshold dynamically. The control module 18 logs and stores data regarding the repeated commands and user feedback for future system improvement. The foregoing paragraphs have been provided by way of general introduction and are not intended to limit the scope of the following claims. The described embodiments, together with further advantages, will be best understood by reference to the following detailed description taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS Figure 1: Block diagram of computing system that illustrates workflow according to an embodiment of the invention DETAILED DESCRIPTION OF THE INVENTION Aspects of the present invention are best understood by reference to the description set forth herein. All the aspects described herein will be better appreciated and understood when considered in conjunction with the following descriptions. It should be understood, however, that the following descriptions, while indicating preferred aspects and numerous specific details thereof, are given by way of illustration only and should not be treated as limitations. Changes and modifications may be made within the scope herein without departing from the spirit and scope thereof, and the present invention herein includes all such modifications. The present invention pertains to a computing system, as depicted by Fig 1. In an embodiment, the computing system is present inside a vehicle and includes various components. These have been explained in detail in subsequent paras. It includes a speech recognition module 10. The speech recognition module 10 is designed to receive an input command in the form of speech and convert the input command into text. The speech recognition module 10 is an integral part of the computing system and plays a pivotal role in processing the user's commands. It uses advanced speech recognition algorithms to accurately convert spoken language into written text. This enables the system to understand and process the user's commands effectively. In an example embodiment, the speech recognition module 10 may be, for example, part of an automotive / vehicle head unit / infotainment system. In some implementations, it may include one or more processors that are configured to perform a transcription, speech recognition, or speech-to-text function. In some implementations, the transcription, speech recognition, or speech-to-text function may be performed on the user prompt processing system (e.g., based on audio data captured by the one or more microphones). The computing system also includes a processing unit 12. This unit is configured to transmit the converted text to a backend system 16 for further processing, for example, a cloud-based platform that supports the voice assistance system with additional processing power / capabilities. The processing unit 12 is a crucial component of the computing system as it carries out the task of transmitting the converted text to the backend system. This step is vital in ensuring that the user's command is processed correctly and the desired action is taken. In an embodiment, the processing unit 12 may include one or more processors (e.g., microprocessors), one or more processing cores, a programmable logic circuit (PLC) or a programmable logic / gate array (PLA / PGA), a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or any other control circuit. In some implementations, the control circuit or computing system may be part of, or may form, a vehicle control unit (also referred to as a vehicle controller) that is embedded or otherwise disposed in the vehicle (e.g., a Mercedes-Benz® car or van). For example, the vehicle controller may be or may include an infotainment system controller (e.g., an infotainment head-unit), a telematics control unit (TCU), an electronic control unit (ECU), a central powertrain controller (CPC), a charging controller, a central exterior and interior controller (CEIC), a zone controller, or any other controller. The system also features a feedback detection module 14. This module is designed to analyze user feedback associated with the response to the input command. It detects one or more indicators of an incorrect response, which can include a facial expression, a gesture, or an emotional state. The feedback detection module uses advanced machine learning algorithms and data analysis techniques to accurately detect and interpret these indicators. This allows the system to understand whether the user is satisfied with the response or not. A control module 18 is another key component of the computing system. This module is configured to retrigger the session when the feedback detection module detects an indicator of an incorrect response. It prompts the user to repeat the input command to initiate a new session. The control module plays a crucial role in ensuring that the system can effectively handle incorrect responses and provide the user with an opportunity to correct their command. The feedback detection module 14 further comprises a deep neural network (DNN) trained to classify user facial expressions as positive, neutral, or negative. The DNN is a type of artificial neural network with multiple layers between the input and output layers. It uses complex algorithms and large amounts of data to learn and make accurate predictions. In this case, the DNN is trained to classify user facial expressions, which can provide valuable insights into the user's emotional state and satisfaction with the system's response. The feedback detection module 14 also uses a convolutional neural network (CNN) to process real-time video data of the user’s facial expressions. CNNs are a type of deep learning model that are particularly effective at processing visual data. They use a variation of multilayer perceptrons designed to require minimal preprocessing. In this case, the CNN is used to analyze real-time video data of the user's facial expressions, providing a more accurate and comprehensive understanding of the user's feedback. The control module 18 can also initiate a different response based on the detected emotional state, such as providing suggestions to clarify the input command. This feature of the control module allows the system to provide a more personalized and effective response to the user's command. It can suggest ways to clarify the input command based on the user's emotional state, enhancing the user experience and increasing the system's effectiveness. The system can detect gesture input using a sensor interface comprising touch-based or non-touch-based input sensors available through vehicle infotainment system. These sensors can accurately detect and interpret the user's gestures, providing another layer of feedback for the system. This allows the system to understand the user's commands and feedback more comprehensively, leading to more accurate responses and a better user experience. The feedback detection module is also configured to detect changes in user feedback over time and update the retriggering threshold dynamically. This feature allows the system to adapt to the user's changing feedback and preferences, ensuring that the system remains effective and user-friendly over time. The control module 18 also logs and stores data regarding the repeated commands and user feedback for future system improvement. This data can provide valuable insights into the user's preferences and the system's performance, which can be used to make future improvements to the system. This ensures that the system continues to evolve and improve over time, providing a consistently high-quality user experience. In conclusion, the present invention provides a comprehensive and effective solution for processing user input and feedback in a speech recognition system. It incorporates advanced technologies and algorithms to accurately process and respond to the user's commands, while also taking into account the user's feedback and emotional state. This results in a system that is not only highly effective but also user-friendly and adaptable. Referring to the block diagram illustrating a workflow for processing user input and feedback in a speech recognition system, the system comprises several interconnected components that work in tandem to improve the user experience with voice assistant (VA) technology. The system initiates with a speech recognition module, which receives an input command in the form of speech from the user. This input command is then converted into text by the speech recognition module. This conversion process utilizes advanced speech-to-text algorithms to ensure accurate transcription of the user's spoken command. The text version of the input command is then transmitted to a backend processing unit 16. This backend processing unit 16 is responsible for analyzing the text command and initiating the appropriate response. The response generated by the backend processing unit is then sent back to the user in the form of an audio response, providing the user with the information or action they requested. In an example, this could be a cloud server that is connected to the vehicle infotainment system. A critical component of this system is the feedback detection module 14. This module is designed to analyze user feedback associated with the response to the input command. The feedback detection module is equipped to detect one or more indicators of an incorrect response. These indicators can include a variety of user reactions, such as a facial expression, a gesture, or an emotional state. In the event that the feedback detection module 14 detects an indicator of an incorrect response, a control module is triggered. This control module is configured to retrigger the session by prompting the user to repeat the input command, thereby initiating a new session. The feedback detection module 14 utilizes advanced machine learning models, such as deep neural networks (DNN) and convolutional neural networks (CNN), for accurate recognition of facial expressions and gestures. These models are trained on large datasets to ensure their accuracy and reliability. In addition to the above components, the system also includes various hardware elements for recognizing facial expressions, emotions, and gestures from the user. For instance, an RGB camera is used for facial expression recognition, while depth sensors like Microsoft Kinect or LiDAR sensors provide depth information for more accurate facial feature tracking and expression analysis. The system is designed to continuously learn and adapt based on user feedback. For example, if the user expresses dissatisfaction with the response provided by the VA, the system will retrigger the session and ask for further inputs from the user. This iterative process allows the system to continuously improve and provide more accurate and personalized responses over time. In an alternate embodiment, the system could also use other types of sensors for gesture recognition, such as infrared sensors or motion tracking cameras. Similarly, other types of machine learning models, such as recurrent neural networks (RNNs) or support vector machines (SVMs), could be used for facial expression and gesture recognition. In addition, the system could also incorporate other types of feedback mechanisms, such as voice tone analysis or sentiment analysis, to further enhance its ability to detect incorrect responses and improve the overall user experience. Overall, the system provides a robust and intelligent solution for improving the accuracy and user experience of voice assistant technology. By incorporating advanced machine learning models, sophisticated sensor technology, and an iterative feedback mechanism, the system is able to detect incorrect responses, retrigger sessions, and continuously learn and adapt based on user feedback. In one embodiment of the present invention, a computing system incorporates a range of modules designed to enhance the functionality of a speech recognition system. The speech recognition module is designed to receive an input command in the form of speech and convert it into text. During this process, the spoken command is accurately transcribed into written text, enabling the system to understand and effectively process the user's commands. From here, the converted text is transmitted to a backend system for further processing by a specifically designed processing unit. This crucial component of the system ensures that the command is analyzed and an appropriate response is generated. The response, based on the submitted input command, is sent back to the user, often in the form of an audio output, providing the user with the requested information or action. However, given the complexities of human language and speech, there may be instances where the generated response does not meet the user's expectations or needs. In such cases, the user may express their dissatisfaction through a variety of nonverbal cues such as facial expressions, gestures, or changes in emotional state. Recognizing and responding to these non-verbal cues forms an integral part of this embodiment, enhancing user experience and minimizing potential misunderstandings. To capture and analyze these non-verbal cues, the system incorporates a feedback detection module. This module is designed to detect one or more indicators of an incorrect or unexpected response. It uses advanced machine learning algorithms and data analysis techniques to accurately detect and interpret these indicators. The feedback detection module can recognize a variety of responses, including facial expressions, gestures or an emotional state, providing a comprehensive understanding of the user's satisfaction with the system's response. In the event that an indicator of an incorrect response is detected, the system triggers a control module. This module is designed to reinitiate the session, prompting the user to repeat the input command and thereby starting a new session. This iterative process ensures that the system can effectively handle incorrect or unexpected responses and provide the user with the opportunity to correct or modify their command. In further embodiments, the feedback detection module incorporates a deep neural network (DNN) and a convolutional neural network (CNN). The DNN, a type of artificial neural network with multiple layers, is trained to classify user facial expressions as positive, neutral, or negative. Meanwhile, the CNN, particularly effective at processing visual data, is used to analyze real-time video data of the user's facial expressions. By combining these two types of neural networks, the system is able to provide a more comprehensive and accurate understanding of the user's feedback. In addition to advanced machine learning models, this embodiment also employs a range of hardware for recognizing facial expressions, emotions, and gestures. For instance, an RGB camera can be used for facial expression recognition, while depth sensors, such as a Microsoft Kinect or LiDAR sensor, provide depth information for more accurate facial feature tracking and expression analysis. Furthermore, these sensors can be utilized for gesture recognition, adding another layer of feedback for the system. An additional embodiment of the present invention enables the system to provide a different response based on the detected emotional state of the user. For example, the system could provide suggestions to clarify the input command. This responsive approach can help enhance the user experience and increase the system's effectiveness. The gesture input can be detected using a sensor interface comprising touch-based or non-touch-based input sensors, adding another layer of interaction to the system. A further embodiment of the invention includes a dynamic threshold setting. The feedback detection module is configured to detect changes in user feedback over time and update the retriggering threshold dynamically. This adaptive feature enables the system to continually update and refine its response mechanism in line with the user's changing feedback and preferences. In another embodiment, the control module logs and stores data regarding the repeated commands and user feedback for future system improvement. The collection of this data provides valuable insights into the user's preferences and the performance of the system, which can be used to drive future enhancements, ensuring a constantly evolving and improving user experience. The present invention provides an effective solution to the existing challenges faced by voice recognition systems. By incorporating user feedback in the form of facial expressions, gestures, and emotional states, the system can provide more accurate and personalized responses, enhancing the user experience. The use of advanced DNN and CNN models, coupled with the incorporation of effective sensor technology, gives the system the ability to learn and adapt from user commands, ensuring continued improvement over time. The dynamic retriggering threshold further enhances the system's user-friendliness and effectiveness. In other embodiments, alternate types of sensors could be used for gesture recognition, such as infrared sensors or motion tracking cameras. Similarly, other types of machine learning models, such as recurrent neural networks (RNNs) or support vector machines (SVMs), could be used for facial expression and gesture recognition. These alternatives provide additional options, expanding the potential application and accessibility of the invention. With this invention, the user experience is vastly improved compared to existing solutions, as it offers the potential for more accurate responses and an iterative feedback mechanism. By incorporating advanced machine learning models, sophisticated sensor technology and a user-centric feedback mechanism, the system ensures enhanced accuracy and a highly personalized user experience. The combination of these features results in a speech recognition system that not only improves the user experience but also presents an opportunity for the system to learn and adapt from user commands, a significant advantage over existing solutions. The described embodiments of the invention are illustrative and not limiting. Further embodiments and modifications that do not depart from the spirit and scope of the invention will be apparent to those of ordinary skill in the art from the detailed description and drawings. Thus, the scope of the invention is not limited by the specific embodiments described and illustrated herein.

Claims

1. A computing system comprising:a speech recognition module configured to receive an input command in the form of speech and convert the input command into text;a processing unit configured to transmit the converted text to a backend system for further processing;a feedback detection module configured to analyze user feedback associated with the response to the input command, the feedback detection module detecting one or more indicators of an incorrect response, wherein the indicators include one or more of a facial expression, a gesture, or an emotional state;a control module configured to retrigger the session when the feedback detection module detects an indicator of an incorrect response, the control module prompting the user to repeat the input command to initiate a new session.

2. The computing system of claim 1, wherein the feedback detection module further comprises a deep neural network (DNN) trained to classify user facial expressions as positive, neutral, or negative.

3. The computing system of claim 1, wherein the feedback detection module uses a convolutional neural network (CNN) to process real-time video data of the user’s facial expressions.

4. The computing system of claim 1, wherein the control module further initiates a different response based on the detected emotional state, such as providing suggestions to clarify the input command.

5. The computing system of claim 1, wherein the gesture input is detected using a sensor interface comprising touch-based or non-touch-based input sensors.

6. The computing system of claim 1, wherein the feedback detection module is configured to detect changes in user feedback over time and update the retriggering threshold dynamically.

7. The computing system of claim 1, wherein the control module logs and stores data regarding the repeated commands and user feedback for future system improvement.

8. A computer-implemented method for processing user input and feedback in a speech recognition system, comprising:receiving, by a speech recognition module, an input command in the form of speech;converting, by the speech recognition module, the input command into text and transmitting the text to a backend processing system;providing a response based on the processed input command;detecting, by a feedback detection module, one or more indicators of an incorrect response, wherein the indicators comprise one or more of: a facial expression, a gesture, or an emotional state of the user;retriggering the session when an incorrect response is detected, including prompting the user to repeat the input command.

9. The method of claim 8, further comprising the step of training a neural network model to classify facial expressions associated with user satisfaction or dissatisfaction.

10. The method of claim 8, wherein detecting an emotional state comprises analyzing real-time video of the user through a convolutional neural network (CNN).

11. The method of claim 8, wherein retriggering the session includes providing the user with options for clarifying or modifying the input command based on the detected feedback.

12. The method of claim 8, wherein detecting gesture-based feedback includes processing input from a nonverbal multi-input interface.

13. The method of claim 8, further comprising dynamically adjusting the threshold for detecting an incorrect response based on the user’s interaction history.

14. The method of claim 8, further comprising storing data from repeated sessions to optimize future processing of similar input commands.

Images