Chatbot system and conversation method based on speech recognition and Rasa framework

Abstract

The present invention relates to a chat robot system and a dialogue method based on speech recognition and Rasa framework, wherein the system comprises a speech service module and an intelligent assistant module; the speech service module comprises a speech recognition unit and a speech synthesis unit, wherein the speech recognition unit is used to recognize input speech information and convert the input speech information into text information; the speech synthesis unit is used to convert the received text information into speech information; the intelligent assistant comprises a language understanding unit and a dialogue management unit, wherein the language understanding unit is used to classify user intentions and extract entities according to text information; the dialogue management unit is used to maintain and update the dialogue state and action selection of the user, respond to the user's input according to the understanding result of the speech understanding unit, and output the text information of the reply. The chat dialogue of the robot is made smoother and the user experience is improved.

Description

Technical Field

[0001] The present invention relates to the field of human-computer dialogue, and in particular to a chat robot system and a dialogue method based on speech recognition and a Rasa framework. Background Art

[0002] Natural language processing (NLP) is a field of computer science, artificial intelligence, and linguistics that focuses on the interaction between computers and human (natural) language. By analyzing grammar, semantics, and pragmatics, the semantic representation of natural language is obtained, with the goal of generating a machine-readable semantic representation of natural language for chatbots.

[0003] With the development of artificial intelligence, the Internet and other technologies, chatbots are applied in multiple fields, including telecommunications, tourism, medical aviation, finance and other fields, with obvious effects. Artificial intelligence technology has broken through the original technical bottleneck of chatbots, and practice has proved that the use of chatbots can not only reduce a large amount of labor costs for enterprises, but also significantly improve work efficiency. At present, chatbots are widely used in a wide range of scenarios, including question and answer, virtual assistants, dialogue, community management, customer robots, medical care, etc. And as chatbots become more and more advanced, the scenarios in which they can be applied or the problems they can handle will increase.

[0004] However, due to the lack of maturity of technology, chatbots cannot completely chat in the same way as humans. Even though there are many platforms on the market that can build chatbots, such as Microsoft's XiaoIce (the best for small talk at present), Apple's Siri, Amazon's Amazon echo, etc., they can all chat, but the overall experience is as follows:

[0005] The chatbot is still a bit stupid. It cannot understand humans and gives irrelevant answers, resulting in its actual use effect being far below public expectations. The public believes that this chatbot is useless.

[0006] Poor understanding of contextual semantics in multiple rounds of interactions leads to low fluency of user experience (industry bottleneck) Summary of the invention

[0007] To this end, it is necessary to provide a chat robot system and method based on speech recognition and Rasa framework to solve the problem that the existing chat robots have poor semantic understanding ability, resulting in low user experience fluency.

[0008] To achieve the above objectives, the inventor provides a chatbot system based on speech recognition and Rasa framework, including a voice service module and an intelligent assistant module;

[0009] The voice service module includes a voice recognition unit and a voice synthesis unit. The voice recognition unit is used to recognize the input voice information and convert the input voice information into text information; the voice synthesis unit is used to convert the received text information into voice information;

[0010] The intelligent assistant includes a language understanding unit and a dialogue management unit. The language understanding unit is based on the RasaNLU framework and is used to classify user intent and extract entities based on text information.

[0011] The dialogue management unit is based on the Rasa Core framework. The dialogue management unit is used to maintain and update the user's dialogue status and action selection, respond to the user's input according to the understanding result of the speech understanding unit, and output the reply text information.

[0012] Further optimized, the speech synthesis unit is specifically used to convert text information into a phoneme sequence, and mark the start and end time and frequency change of each phoneme, and generate speech information according to the phoneme sequence, the start and end time and frequency change of the phoneme.

[0013] Further optimized, the speech understanding unit includes a word segmentation component, an entity extraction component, a feature extraction component and an intention recognition component;

[0014] The word segmentation component is used to segment sentences in the input text information into individual words;

[0015] The entity extraction component is used to extract set keywords based on the segmented words;

[0016] The feature extraction component is used to extract the features of the sentence according to the segmented words;

[0017] The intent recognition component is used to recognize intent based on the extracted features.

[0018] Further optimized, the speech understanding unit also includes an initialization component, which is used to initialize the content required for the work of the word segmentation component, the entity extraction component, the feature extraction component and the intent recognition component.

[0019] Further optimization includes a business customization service module, which is used to set corresponding business behavior interfaces according to actual business needs, and the business behavior interfaces include a chat interface, a voice interface and a ticket booking interface.

[0020] Further optimization also includes a tool management module, which is used for content management, story management, offline training, model management and behavior management.

[0021] Another technical solution is also provided: a chatbot dialogue method based on speech recognition and Rasa framework, comprising the following steps:

[0022] The voice recognition unit recognizes the input voice information and converts the input voice information into text information;

[0023] The language understanding unit performs user intent classification and entity extraction based on text information;

[0024] The dialogue management unit updates the user's dialogue status and action selection, responds according to the user's intention and the extracted entities, and outputs the response text information;

[0025] The speech synthesis unit converts the response text information into speech information.

[0026] Further optimized, the step of "the speech synthesis unit converts the response text information into speech information" specifically includes the following steps:

[0027] The speech synthesis unit converts text information into a phoneme sequence, marks the start and end time of each phoneme and the frequency change, and generates speech information based on the phoneme sequence, the start and end time of the phoneme and the frequency change.

[0028] Different from the existing technology, the above technical solution converts the voice information input by the user into text information through the voice recognition unit, and then understands the text information through the voice understanding unit based on the Rasa NLU framework to obtain the user's intention classification and entity extraction; then the dialogue management unit updates the user's dialogue status and action selection, responds to the user's input, outputs the response text information, and then converts the response text information into voice information output through the voice synthesis unit to complete the voice chat with the user. By utilizing the two core technologies of speech recognition technology and Rasa open source machine learning framework, the Rasa framework has the characteristics of easy-to-use, easy-to-train and reusable pattern matching methods and search methods; at the same time, because Rasa nlu provides a pipeline approach, Rasa code provides complete dialogue management, which greatly improves the scalability and application field, while improving the accurate reading of user intentions, making the robot chat dialogue smoother and improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A structural diagram of a chatbot system based on speech recognition and Rasa framework described in a specific implementation method;

[0030] Figure 2 A structural diagram of a chatbot system based on speech recognition and Rasa framework described in a specific implementation method;

[0031] Figure 3 A schematic diagram of a flow chart of speech recognition described in a specific implementation method;

[0032] Figure 4 A flowchart of a chatbot dialogue method based on speech recognition and Rasa framework described in a specific implementation method.

[0033] Description of reference numerals:

[0034] 111. speech recognition unit, 112. speech synthesis unit;

[0035] 121. Language comprehension unit, 122. Dialogue management unit;

[0036] 130. Business customization service module, 131. Chat interface, 132. Voice interface, 133. Ticket booking interface. DETAILED DESCRIPTION

[0037] In order to explain the technical content, structural features, achieved objectives and effects of the technical solution in detail, the following is a detailed description in conjunction with specific embodiments and accompanying drawings.

[0038] See also Figure 1-3 ,This embodiment provides a chat robot system based on speech recognition and Rasa framework, including a voice service module and an intelligent assistant module;

[0039] The voice service module includes a voice recognition unit 111 and a voice synthesis unit 112. The voice recognition unit 111 is used to recognize the input voice information and convert the input voice information into text information; the voice synthesis unit 112 is used to convert the received text information into voice information;

[0040] The intelligent assistant includes a language understanding unit 121 and a dialogue management unit 122. The language understanding unit 121 is based on the Rasa NLU framework and is used to classify user intent and extract entities based on text information.

[0041] The dialogue management unit 122 is based on the Rasa Core framework. The dialogue management unit 122 is used to maintain and update the user's dialogue status and action selection, respond to the user's input according to the understanding result of the speech understanding unit, and output the reply text information.

[0042] See also Figure 3Speech recognition (Automatic Speech Recognition) is a research object based on speech. Speech recognition technology is a technology that allows machines to convert speech signals into corresponding text or commands through the process of recognition and understanding. Speech recognition is a cross-disciplinary subject with a wide range of aspects. It has a very close relationship with acoustics, phonetics, linguistics, information theory, pattern recognition theory, neurobiology and other disciplines. The speech recognition of the speech recognition unit 111 includes three basic units: feature extraction, pattern matching, and reference pattern library. Its basic structure is as follows: Figure 3 As shown in the figure, when voice is input, the input voice is first preprocessed, and then the voice features are extracted, and the template required for voice recognition is established on this basis. During the recognition process, the computer will compare the voice template stored in the computer with the features of the input voice signal according to the voice recognition model, and find a series of optimal templates that match the input voice according to certain search and matching strategies. Then, according to the definition of this template, the computer's recognition result can be given by looking up the table.

[0043] The intelligent assistant module includes language understanding (Rasa NLU) and dialogue management (Rasa Core). During the human-computer dialogue, the user enters the corresponding text information (or speech recognition results), and the specific language understanding module performs user intent classification, entity extraction and other processes. At the same time, the dialogue management module tracks the user dialogue status, selects the corresponding candidate actions according to certain strategies, and executes the corresponding actions (including: information reply and business behavior execution)

[0044] The voice information input by the user is converted into text information through the voice recognition unit 111, and then the text information is understood by the voice understanding unit based on the Rasa NLU framework to obtain the user's intention classification and entity extraction; then the user's dialogue status and action selection are updated through the dialogue management unit 122, responding to the user's input, outputting the response text information, and then the response text information is converted into voice information output through the voice synthesis unit 112 to complete the voice chat with the user. By utilizing the two core technologies of speech recognition technology and Rasa open source machine learning framework, the Rasa framework has the characteristics of easy-to-use, easy-to-train and reusable pattern matching methods and search methods; at the same time, because Rasa nlu provides a pipeline-style approach, Rasa code provides complete dialogue management, which greatly improves the scalability and application field, while improving the accurate reading of user intentions, making the robot chat dialogue smoother and improving the user experience.

[0045] In this embodiment, the user can send voice information to the chat robot system through the client, such as the web terminal or the APP mobile terminal, and the chat robot can also send the voice information of the conversation to the client; the user can also directly input voice information to the chat robot through the voice device on the chat robot system, and then the chat robot communicates with the user through its own voice device.

[0046] In this embodiment, the speech synthesis unit 112 is specifically used to convert text information into a phoneme sequence, mark the start and end time of each phoneme and the frequency change, and generate speech information according to the phoneme sequence, the start and end time of the phoneme and the frequency change. Speech synthesis TTS (Text-To-Speech) is generally divided into two steps:

[0047] First, text processing: mainly converting text information into phoneme sequences and marking the start and end time of each phoneme, frequency change and other information. First, the text information is segmented, and then the text information is converted into sentences consisting of only words. Then the sentences are annotated with information that is helpful for speech synthesis, such as the phoneme level (previous phoneme / next phoneme), syllable level (the number of syllables in the word), and word level (part of speech / position in the sentence).

[0048] 2. Speech synthesis: It mainly generates speech information based on the phoneme sequence and the start and end time and frequency changes of the marked phonemes. There are three main methods: splicing method, parameter method and vocal tract simulation method.

[0049] In this embodiment, Rasa NLU is an open source natural language processing tool for intent classification, response retrieval, and entity extraction in conversational robots. The Rasa NLU framework completes intent recognition through managed components. However, this recognition process is not a one-step process and requires the division of labor and cooperation of multiple components to complete. The components are like assembly line operations. Each component processes the input data and outputs the processing results for use by other components or as the final output. In this way, different processing methods can be provided for each step. Specifically, the speech understanding unit includes a tokenizer, an entity extraction component, a feature extraction component, and an intent recognition component (classifier);

[0050] The word segmentation component is used to segment sentences in the input text information into separate words; the Jieba word segmentation device is used to perform Chinese word segmentation.

[0051] The entity extraction component is used to extract set keywords based on the segmented words; MitieEntityExtractor is used to extract entities in the sentence.

[0052] The feature extraction component is used to extract the features of the sentence according to the segmented words; RegexFeaturizer and MitieFeaturizer are used for feature extraction.

[0053] The intent recognition component is used to recognize intent based on the extracted features and use the Sklearn intent classifier to classify intent.

[0054] The speech understanding unit further includes an initialization component, which is used to initialize the content required for the work of a word segmentation component, an entity extraction component, a feature extraction component, and an intent recognition component.

[0055] In this embodiment, Rasa Core is the dialogue management unit 122 provided by the Rasa framework. It is similar to the brain of the chatbot. Its main task is to maintain and update the dialogue state and action selection, and then respond to the user's input. The so-called dialogue state is a representation of chat data that can be processed by a machine. The dialogue state contains all information that may affect the next decision, such as the output of the natural language understanding module, the characteristics of the user, etc.; the so-called action selection refers to selecting the next appropriate action based on the current dialogue state, such as asking the user for additional information, executing the action required by the user, etc. To give a specific example, the user says "help me book a plane ticket", at this time the dialogue state includes the output of the natural language understanding module, the user's location, historical behavior and other characteristics. In this state, the next action of the system may be:

[0056] 1. Ask the user which city they want to reach, such as "Which city do you want to book a flight to?"

[0057] 2. Ask the user to confirm the departure city (which can be obtained through positioning) and the destination city, such as "Are you looking to book a flight from Beijing to Shanghai?"

[0058] 3. Confirm the departure date for the user, such as "Which day do you want to book a flight?"

[0059] 4. Confirm the flight ticket with the user, such as "Can I book Xiamen Airlines MF8555 for you?"

[0060] 5. Book tickets directly for users.

[0061] In this embodiment, a business customization service module 130 is also included. The business customization service module 130 is used to set corresponding business behavior interfaces according to actual business needs. The business behavior interfaces include a chat interface 131 , a voice interface 132 and a ticket booking interface 133 .

[0062] In this embodiment, a tool management module is also included, and the tool management module is used for content management, story management, offline training, model management, and behavior management. The tool management module can realize the online collection, online annotation, and online editing of the training data required by the intelligent assistant module, and realize the online training, publishing, and management of the model. At the same time, the business customization service module 130 is integrated online. This makes the chatbot system suitable for various business scenarios, simplifies system operation and maintenance, and continuously improves the accuracy and recall rate of human-computer dialogue. The specific implementation method is as follows:

[0063] (1) Content management:

[0064] Content management is mainly used for entering, managing and marking content in the dialogue system. Content types can be: text, graphics and files. Functions include: content classification management, content addition, editing, deletion and file import.

[0065] For the input content information, the backend service will perform the following steps of data preprocessing:

[0066]

[0067]

[0068] Step 1: Get the title and content using Excel parsing, Tika extraction and other technologies.

[0069] Step 2: Data cleaning, using techniques such as Html tag filtering, regular matching, special character filtering, and removal of stop words.

[0070] Step 3: Fill in the gaps in titles. For records without titles after steps 1 and 2, the seq2seq model is used to automatically generate titles using the cleaned text information.

[0071] Step 4: Manual labeling (this step is not necessary). The system provides many question templates. Manually select the template for the data and set the label (the default is the title if not set). The system will automatically generate questions using the template matching method.

[0072] (2) Story Management:

[0073] Story management is mainly used to arrange story scenes for dialogue management, so that the system can automatically select the corresponding response strategy based on user intentions. Functions include: adding stories, online editing, deleting, and importing.

[0074] (3) Online training:

[0075] Online training is mainly used for online training of Rasa models. After completing steps (1) and (2), the system is ready for online training. During online training, the system will automatically generate Rasa training data based on content management data, and then start model training. The training results will be automatically saved to the models directory, and the model data will be recorded at the same time. Rasa training data includes: version, nlu, stories, and rules. The details are as follows:

[0076] Nlu:

[0077] NLU training data stores structured information about user messages. The goal of NLU (Natural Language Understanding) is to extract structured information from user messages. This typically includes the user's intent and any entities their messages contain. Additional information such as regular expressions and lookup tables can be added to the training data to help the model correctly identify intent and entities.

[0078] stories:

[0079] Stories are a type of training data used to train assistant dialogue management models. Stories can be used to train models that can generalize to unseen dialogue paths. Stories are representations of conversations between users and AI assistants, converted into a specific format where user inputs are represented as intents (and entities when necessary), and the assistant's responses and actions are represented as action names.

[0080] (4) Model management:

[0081] Model management is mainly used to manage models generated by training, including functions such as model deletion and publishing.

[0082] (5) Action management:

[0083] Action management is mainly used to manage the Action interface information customized by the business module, including: the URL for calling the service, the action name, and the action unique key. Functions include: Action registration, editing, and deletion.

[0084] See also Figure 4 In another embodiment, a chatbot dialogue method based on speech recognition and Rasa framework is applied to the above-mentioned chatbot system based on speech recognition and Rasa framework, and the dialogue method includes the following steps:

[0085] S310: recognizing the input voice information through the voice recognition unit, and converting the input voice information into text information;

[0086] S320: Classifying user intent and extracting entities based on text information through a language understanding unit;

[0087] S330: The dialog management unit updates the user's dialog state and action selection, responds according to the user's intention and the extracted entity, and outputs text information of the response;

[0088] S340: The speech synthesis unit converts the response text information into speech information.

[0089] The voice information input by the user is converted into text information through the voice recognition unit, and then the text information is understood by the voice understanding unit based on the RasaNLU framework to obtain the user's intention classification and entity extraction; then the user's dialogue status and action selection are updated through the dialogue management unit, responding to the user's input, outputting the response text information, and then the response text information is converted into voice information output through the speech synthesis unit to complete the voice chat with the user. By utilizing the two core technologies of speech recognition technology and Rasa open source machine learning framework, the Rasa framework has the characteristics of easy-to-use, easy-to-train and reusable pattern matching methods and search methods; at the same time, because Rasa nlu provides a pipeline approach, Rasa code provides complete dialogue management, which greatly improves the scalability and application field, while improving the accurate reading of user intentions, making the robot chat dialogue smoother and improving the user experience.

[0090] In this embodiment, the step of "the speech synthesis unit converts the response text information into speech information" specifically includes the following steps:

[0091] The speech synthesis unit converts text information into a phoneme sequence, marks the start and end time of each phoneme and the frequency change, and generates speech information based on the phoneme sequence, the start and end time of the phoneme and the frequency change.

[0092] Speech synthesis TTS (Text-To-Speech) is generally divided into two steps:

[0093] First, text processing: mainly converting text information into phoneme sequences and marking the start and end time of each phoneme, frequency change and other information. First, the text information is segmented, and then the text information is converted into sentences consisting of only words. Then the sentences are annotated with information that is helpful for speech synthesis, such as the phoneme level (previous phoneme / next phoneme), syllable level (the number of syllables in the word), and word level (part of speech / position in the sentence).

[0094] 2. Speech synthesis: It mainly generates speech information based on the phoneme sequence and the start and end time and frequency changes of the marked phonemes. There are three main methods: splicing method, parameter method and vocal tract simulation method.

[0095] It should be noted that, although the above embodiments have been described in this article, the patent protection scope of the present invention is not limited thereby. Therefore, based on the innovative concept of the present invention, changes and modifications made to the embodiments described herein, or equivalent structures or equivalent process changes made using the contents of the present invention specification and drawings, directly or indirectly applying the above technical solutions to other related technical fields, are all included in the patent protection scope of the present invention.

Claims

1. A chatbot system based on speech recognition and Rasa framework, characterized in that: Including voice service module, intelligent assistant module, business customization service module and tool management module; The voice service module includes a voice recognition unit and a voice synthesis unit. The voice recognition unit is used to recognize the input voice information and convert the input voice information into text information; the voice synthesis unit is used to convert the received text information into voice information; The intelligent assistant module includes a language understanding unit and a dialogue management unit. The language understanding unit is based on the RasaNLU framework and is used to classify user intent and extract entities based on text information. The dialogue management unit is based on the Rasa Core framework, and is used to maintain and update the user's dialogue status and action selection, respond to the user's input according to the understanding result of the speech understanding unit, and output the reply text information; The business customization service module is used to set corresponding business behavior interfaces according to actual business needs, and the business behavior interfaces include a chat interface, a voice interface and a ticket booking interface; The tool management module is used for content management, story management, online training, model management and action management; the tool management module realizes the online collection, online annotation and online editing of the training data required by the intelligent assistant module, and realizes the online training, publishing and management of the model; at the same time, the tool management module realizes the online integration of the business customization service module, so that the chatbot system can be suitable for various business scenarios; Online training is used for online training of Rasa models. During online training, the system will automatically generate Rasa training data based on content management data, and then start model training. The training results are automatically saved to the models directory, and the model data is recorded at the same time. Rasa training data includes: version, nlu, stories, and rules. The nlu training data stores structured information about user messages. The goal of nlu is to extract structured information from user messages. The goal of nlu includes the user's intent and any entities contained in the user's message, and adds additional information of regular expressions and lookup tables to the training data to help the model correctly identify intent and entities. Stories are used to train models that can generalize to unseen dialogue paths. Stories are representations of conversations between users and AI assistants, converted into a specific format where user inputs are represented as intents and entities when necessary, while the assistant’s responses and actions are represented as action names. Action management is used to manage the Action interface information customized by the business customization service module.

2. The chat robot system based on speech recognition and Rasa framework according to claim 1, characterized in that: The speech synthesis unit is specifically used to convert text information into a phoneme sequence, mark the start and end time of each phoneme and the frequency change, and generate speech information according to the phoneme sequence, the start and end time of the phoneme and the frequency change.

3. The chat robot system based on speech recognition and Rasa framework according to claim 1 is characterized in that: The speech understanding unit includes a word segmentation component, an entity extraction component, a feature extraction component and an intention recognition component; The word segmentation component is used to segment sentences in the input text information into individual words; The entity extraction component is used to extract set keywords based on the segmented words; The feature extraction component is used to extract the features of the sentence according to the segmented words; The intent recognition component is used to recognize intent based on the extracted features.

4. The chat robot system based on speech recognition and Rasa framework according to claim 3 is characterized in that: The speech understanding unit also includes an initialization component, which is used to initialize the content required for the operation of a word segmentation component, an entity extraction component, a feature extraction component, and an intent recognition component.

5. The chat robot system based on speech recognition and Rasa framework according to claim 1, characterized in that: It also includes a tool management module, which is used for content management, story management, offline training, model management, and behavior management.

6. A chatbot dialogue method based on speech recognition and Rasa framework, characterized in that: The method is implemented based on the chatbot system based on speech recognition and Rasa framework according to any one of claims 1 to 5, and comprises the following steps: The voice recognition unit recognizes the input voice information and converts the input voice information into text information; The language understanding unit performs user intent classification and entity extraction based on text information; The dialogue management unit updates the user's dialogue status and action selection, responds according to the user's intention and the extracted entities, and outputs the response text information; The speech synthesis unit converts the response text information into speech information; The step of "the speech synthesis unit converts the response text information into speech information" specifically includes the following steps: The speech synthesis unit converts text information into a phoneme sequence, marks the start and end time of each phoneme and the frequency change, and generates speech information based on the phoneme sequence, the start and end time of the phoneme and the frequency change.

Images