Station service robot voice question-answering system based on passenger travel intention recognition
By introducing corpus construction, WaveNet, Transformer and Mini-Omni models into station service robots, the problem of accurate and inefficient speech recognition in complex environments is solved, and efficient passenger travel intention recognition and Smart Q&A service improves user experience and multilingual support.
Patent Information
- Application Number
- CN202411568678.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-05
- Publication Date
- 2025-01-10
AI Technical Summary
Traditional station service robots are incompetent in dealing with complex or non-standard passenger problems, and due to the complexity of the station environment, such as noisy crowds and diverse dialects, the accuracy and efficiency of speech recognition are limited.
A station service robot voice Q&A system based on passenger travel intention recognition is designed, including a corpus construction module, a travel Q&A voice trigger word identification module, a passenger travel intention recognition module and a multi-round Q&A generation module, using WaveNet, Transformer and Mini-Omni The model performs speech separation, recognition and intent understanding, and combines cloud computing and intelligent interfaces to provide intelligent services.
It realizes accurate identification of passengers' travel intentions in noisy environments, provides convenient, intelligent and personalized travel services, improves the accuracy and user experience of voice recognition, and supports multilingual and diverse needs.
Smart Images
Figure CN119296529A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of speech processing, and in particular to a station service robot speech question-answering system based on passenger travel intention recognition. Background Art
[0002] With the development of intelligent technology, station service robots are playing an increasingly important role in providing passenger services. Traditional station service robots mainly rely on preset question-and-answer libraries to respond to passengers' inquiries, but this approach is inadequate when dealing with complex or non-standard passenger questions. In addition, due to the complexity of the station environment, such as noisy crowds and diverse dialects, the accuracy and efficiency of speech recognition have become key factors restricting the performance of service robots. Summary of the invention
[0003] 1. Technical issues to be solved
[0004] In view of the shortcomings of the existing technology, the present invention provides a station service robot voice question and answer system based on passenger travel intention recognition, which has the advantages of providing intelligent services and prompts according to passengers' needs, and solves the above-mentioned technical problems.
[0005] (II) Technical solution
[0006] To achieve the above-mentioned object, the present invention provides the following technical solutions: a station service robot voice question-answering system based on passenger travel intention recognition, comprising a corpus construction module, a travel question-answering voice trigger word discrimination module, a passenger travel intention recognition module, a multi-round question-answering generation module and a voice question-answering interface integrated with cloud computing;
[0007] The corpus construction module is used to construct a speech corpus and a question-and-answer corpus. The speech corpus stores surveillance videos in the station, self-portrait videos of passengers, radio recordings of the station, and related interview documentaries. The speech corpus includes question-and-answer scenarios of route inquiries, ticket purchase instructions, and carriage conditions during passengers' station travel. The question-and-answer corpus is provided with conversation data about passengers' station travel obtained from the official website of the station and travel forums on social media platforms. The conversation data about station travel includes question-and-answer conversations between passengers and answers to passengers' questions by station staff.
[0008] The travel question and answer voice trigger word discrimination module is equipped with a trained WaveNet model, which is used to separate multi-person audio and extract voice for recognition, and send the recognized voice to the passenger travel intention recognition module;
[0009] The passenger travel intention recognition module converts speech into natural language segments, understands the contextual meaning of the natural language segments, and outputs the passenger travel intention recognition result;
[0010] The travel question and answer voice trigger word identification module sends the recognized voice to the multi-round question and answer generation module to extract the voice feature vector, and transmits it to the voice dialogue model Mini-Omni to decode the voice feature vector, generate the corresponding text token and voice token, and perform fusion output.
[0011] As a preferred technical solution of the present invention, the specific execution steps of the travel question and answer voice trigger word identification module are as follows:
[0012] A1. After receiving a mixed audio signal containing multiple passengers’ voices, it is input into the trained WaveNet model to separate the individual audio data of each passenger;
[0013] A2. Use Soundex method to convert speech into corresponding sound code;
[0014] A3. Compare the encoding result with the pre-stored encoding in the voice corpus, calculate the Levenshtein distance between the query encoding and the encoding in the library, convert it into a similarity score, find the voice segment most similar to the passenger voiceprint information, and output it as the search result.
[0015] As a preferred technical solution of the present invention, the passenger travel intention recognition module uses a Transformer model to convert speech into natural language segments and understand the contextual meaning of the natural language segments.
[0016] Compared with the prior art, the present invention provides a station service robot voice question-answering system based on passenger travel intention recognition, which has the following beneficial effects:
[0017] The present invention provides users with more convenient, intelligent and personalized travel services through the advantages of comprehensive corpus construction, accurate voice triggering and recognition, efficient intent recognition and question and answer generation, intelligent interface and multi-language support. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] Figure 1 It is a schematic diagram of the system framework of the present invention;
[0019] Figure 2 Schematic diagram of the WaveNet model of the present invention.
[0020] Figure 3 This is a schematic diagram of Mini-Omni speech signal processing of the present invention;
[0021] Figure 4 Schematic diagram of the voice interaction interface integrating cloud computing in the present invention. DETAILED DESCRIPTION
[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0023] See also Figure 1-4 , a station service robot voice question and answer system based on passenger travel intention recognition, including a corpus construction module, a travel question and answer voice trigger word discrimination module, a passenger travel intention recognition module, a multi-round question and answer generation module and a voice question and answer interface integrated with cloud computing;
[0024] The corpus construction module is used to construct a speech corpus and a question-and-answer corpus. The speech corpus stores surveillance videos in the station, self-portrait videos of passengers, radio recordings of the station, and related interview documentaries. The speech corpus includes question-and-answer scenarios of passengers’ route inquiries, ticket purchase instructions, and carriage conditions during their station trips. The question-and-answer corpus contains conversation data about passengers’ station trips obtained from the official website of the station and travel forums on social media platforms. The conversation data about station trips includes question-and-answer conversations between passengers and answers given by station staff to passengers’ questions.
[0025] When constructing the speech corpus, we first collect speech clips closely related to station travel from various audio and video resources. These resources include surveillance videos in the station, self-portrait videos of passengers, radio recordings of the station, and related interviews or documentaries. Subsequently, these speech clips are carefully screened and sorted to ensure that they are not only related to station travel, but also clear in sound quality and easy to understand. On this basis, the speech clips are classified according to dimensions such as gender, age, region and language, aiming to build a diverse speech corpus. The speech clips in the speech corpus can fully reflect various question-and-answer scenarios in passengers' station travel, such as route inquiries, ticket purchase instructions, carriage conditions, etc. At the same time, specific trigger information is set for each speech clip, which will serve as the input of the passenger travel intention recognition module, providing a basis for subsequent natural language processing and intention recognition.
[0026] In terms of constructing a passenger travel question and answer corpus, conversation data related to station travel were extracted from the official websites of each station, social media platforms, and related travel forums. These data cover question and answer exchanges between passengers and the answers given to passengers by station staff. Subsequently, the crawled natural language sentences were segmented using the conditional random field method to decompose them into sentence components. In order to deeply understand the semantics of these sentence components, they were annotated with semantic information based on the CLIP model. The CLIP model uses the correspondence between images and texts to link sentence components with specific semantic concepts. Finally, the annotated sentence components were reintegrated in the form of question and answer dialogues to form a passenger travel question and answer corpus that comprehensively reflects the station travel question and answer scenarios and semantic needs;
[0027] The travel question-and-answer voice trigger word recognition module is equipped with a trained WaveNet model, which is used to separate multi-person audio and extract voice for recognition. The recognized voice is sent to the passenger travel intention recognition module.
[0028] When the robot receives real-time voice input from passengers, it first determines whether there are multiple voices. This step is crucial because multiple people speaking at the same time significantly increases the difficulty of voice recognition. If it detects that the voice data contains multiple voices, the robot will use the advanced audio separation method WaveNet to separate the individual audio data of different passengers from the mixed voice, such as Figure 2 As shown in the figure, the robot first receives a mixed audio signal containing multiple passenger voices. This signal is input into the pre-trained WaveNet model. The model uses its internal network blocks to gradually analyze the features in the audio signal through causal convolution and dilated convolution operations. These features include the pitch, timbre, and speaking speed of different passengers' voices, which are used by the model to distinguish different sound sources. As the audio signal propagates forward in the model, each network block outputs an intermediate representation, which is accumulated through jump connections and finally passed to the Softmax layer. The Softmax layer calculates the probability distribution of the voices of different passengers at each time point based on these accumulated representations. Finally, the robot uses this probability distribution information to separate the individual audio data of each passenger from the mixed audio. This process enables the robot to accurately identify and separate the voices of different passengers in a noisy environment, providing high-quality input data for subsequent tasks such as speech recognition and natural language processing.
[0029] After successfully separating the individual audio data of each passenger, the robot will further extract the voiceprint information from these audio data. Voiceprint information is a unique biometric feature that can help the robot identify different speakers. Next, the robot uses the Soundex method to convert the voice into similar sound codes to facilitate search and matching. First, the model converts the voice into the corresponding pinyin and removes the tone symbols, and then converts it to uppercase. Next, the first letter of the pinyin is extracted as the start of the encoding, and traverses from the second letter, converting the consonant letters into corresponding numbers, while ignoring the vowels. The generated encoding result must be limited to four characters, and the insufficient part is padded with zeros. After that, the encoding result is compared with the pre-stored encoding in the voice corpus, and the Leven difference between the query encoding and the encoding in the library is calculated. Shtein distance is converted into a similarity score to find the voice segment that is most similar to the passenger's voiceprint information and output it as the search result. The robot uses the Soundex method to search for sounds similar to the station travel question and answer voice segments pre-stored in the voice corpus from the voiceprint information of each passenger. This voice corpus contains various station travel-related question and answer scenarios and is an important basis for the robot to perform intention recognition. Specifically, the robot searches for sounds similar to the station travel question and answer voice segments pre-stored in the voice corpus from the voiceprint information of each passenger to calculate the similarity. If the similarity reaches the preset threshold, it means that the current passenger's voice is highly similar to a question and answer voice segment in the corpus. At this time, the robot will extract the passenger's voice voiceprint information in real time and trigger the station passenger travel intention recognition module;
[0030] The passenger travel intention recognition module converts speech into natural language segments, understands the contextual meaning of the natural language segments, and outputs the passenger travel intention recognition results;
[0031] In order to reduce the difficulty of directly inferring the content of speech, the Transformer model is used to convert the speech of the question into natural language, and understand the contextual meaning of the natural language fragment, so as to achieve more accurate semantic understanding of travel intention. Finally, the extracted features are fused by time step and sent to the long short-term memory neural network for feature representation learning, and then the passenger travel intention recognition results are output. In this process, the system can also use natural language processing techniques, such as part-of-speech tagging and named entity recognition, to further extract key information from the text and provide richer data support for subsequent semantic understanding. In order to further improve the accuracy of semantic understanding, the system can also combine knowledge graphs and ontologies to associate entities and concepts in the text with external knowledge bases to obtain more comprehensive semantic information. This module uses advanced natural language processing and machine learning technologies to accurately identify passengers' travel intentions, such as querying routes, ticket purchase methods, or car conditions;
[0032] The travel question-and-answer voice trigger word identification module sends the recognized voice to the multi-round question-and-answer generation module to extract the voice feature vector, and transmits it to the voice dialogue model Mini-Omni to decode the voice feature vector, generate the corresponding text token and voice token, and fuse them for output;
[0033] The real-time speech of passengers containing their travel intentions is first fed into the pre-trained large speech model Whisper, which performs in-depth analysis on the speech and extracts a series of continuous and information-rich feature vectors. These feature vectors are the basis for subsequent processing, and they can capture key information in the speech, such as intonation, speaking speed, and volume.
[0034] This application makes full use of the well-built passenger travel question and answer corpus, which covers a rich range of travel question and answer scenarios and semantic needs, to further train and optimize the RoBERTa model. In this way, the RoBERTa model can deeply learn and master the patterns and associations in question and answer pairs, and can provide more accurate and natural answers in actual conversations. When the passenger's travel intention is input into the optimized RoBERTa model, the model will output a more accurate feature vector that is more in line with the user's actual travel intention, significantly improving the performance of the dialogue system and user experience.
[0035] These feature vectors are then passed to the large voice dialogue model Mini-Omni. Mini-Omni's task is to align voice information with text information, that is, to establish a mapping relationship between voice feature vectors and corresponding text content. At this stage, Mini-Omni will use its deep learning capabilities to decode the voice feature vectors and generate corresponding text tokens. At the same time, Mini-Omni will further decode the voice feature vectors through the audio decoding head to generate voice tokens.
[0036] like Figure 3 As shown in Figure 1, this system converts continuous speech signals into discrete speech tokens and uses these tokens for modeling. Set L = (l i ∈T text |i=1,...,t text ) is the text utterance of the question-answering semantic library T, and its probability can be expressed as For a continuous speech signal, it can be converted into discrete speech tags N = (n i ∈T nst |i=1,...,t text ), where T nst is a vocabulary of discrete speech tokens. These discrete speech tokens can be modeled in T just like text.
[0037] By combining text and speech in a new vocabulary T toxt =T test ∪T nst In , the probabilities of speech and text tokens can be modeled. but Where K represents a discrete speech token D or a text token L, or various combinations of L and D. represents the cumulative multiplication of x elements. For the audio and text tags generated simultaneously, the negative log-likelihood loss can be expressed as:
[0038]
[0039] Where X and Y are the text-audio output pairs in the training corpus W, a is the number of training samples, and Q j is the input condition of the jth sample, b j is a sample X j and Y j The maximum number of markers, X i,j and Y i,j represent the i-th text tag and audio tag of the j-th sample, respectively.
[0040] In the output stage of Mini-Omni, text tokens and speech tokens are generated at each step. These tokens are then fed into the feature fusion module, which fuses the audio representation (generated by the speech token) and the text representation (generated by the text token) to form a unified feature vector that combines speech and text information. This feature vector contains not only the acoustic features of the speech, but also the semantic information of the text, providing rich context for subsequent processing. Next, the fused feature vector is used as the input of the next step and continues to participate in the iterative processing of Mini-Omni. In this process, Mini-Omni will gradually build a more accurate and rich question-answering context based on the output of the previous step and the current input.
[0041] At the same time, the voice tokens generated during the multi-round question and answer generation process will be streamed into the multi-scale neural audio codec. The codec uses a multi-scale neural network structure to efficiently and accurately encode and decode the input voice tokens. Through this process, the codec can generate high-quality voice output and achieve low-latency multi-round voice question and answer. In each round of question and answer, the model will quickly generate the corresponding answer voice based on the question and answer content of the previous round and the input voice of the current round. In this process, the model not only utilizes the acoustic features of the voice, but also fully combines the semantic information of the text, thereby providing users with a smooth and natural interactive experience;
[0042] Voice Q&A interface: To improve the passenger travel experience, the voice Q&A interface integrates cloud computing and intelligent prompt systems. In addition, the voice Q&A system also supports multiple languages and can automatically switch the interface language according to the passenger's language preference to meet the needs of passengers in different regions. By building a trained voice model, you can test it. The results are as follows: Figure 3 As shown, the large voice model built through the above steps can be directly deployed on various edge devices, such as the CSK6011A integrated development board with microphone and speaker. Through the wake-up word, you can have a conversation with the voice assistant to process the passengers' real-time questions and output the final answer. In this way, the present invention can be applied to various scenarios, such as smart homes, vehicle-mounted systems, smart office equipment, station service robots, etc., to provide passengers with a convenient voice question and answer experience, and integrate the voice interaction interface of cloud computing: By building a trained large voice model, you can test it, and the results are as follows: Figure 4 As shown;
[0043] The present invention has the following advantages:
[0044] (1) Comprehensive corpus construction: By collecting speech and conversation related to station travel from audio, video and web data, a corpus containing diverse speech segments was constructed. This not only covers passengers of different genders, ages, regions and languages, but also uses conditional random fields and CLIP models to segment and semantically annotate sentence components, ensuring the richness and accuracy of the corpus, providing a solid foundation for subsequent intent recognition and question-answer generation.
[0045] (2) Accurate voice triggering and recognition: The system can receive voice input from passengers and effectively judge and process situations where multiple people are speaking. By using the WaveNet method to separate audio data and extract voiceprint information, and using the Soundex method for similarity judgment, the system can accurately identify passengers' travel intentions in noisy environments, improving the accuracy and reliability of recognition.
[0046] (3) Efficient intent recognition and question-answer generation: The Transformer model is used to convert speech into natural language, and the pre-trained RoBERTa model is used to understand the contextual meaning, achieving more accurate semantic understanding of travel intentions. At the same time, the multi-round question-answer generation module achieves low-latency, high-quality voice question-answer interaction through the collaborative work of Whisper, Mini-Omni, and multi-scale neural audio codec, improving the user experience.
[0047] (4) Intelligent interface and multi-language support: The voice Q&A interface integrates cloud computing and intelligent prompt systems, and can provide intelligent services and prompts based on passengers’ needs. In addition, the system also supports multiple languages and can automatically switch the interface language based on the passenger’s language preference, meeting the diverse needs of passengers in different regions.
[0048] (5) Innovative technology integration: The system combines advanced speech processing, natural language understanding and cloud computing technologies to achieve a complete process from speech input to intent recognition and question and answer generation. This innovative technology integration not only improves the performance of the system, but also provides users with more convenient and intelligent travel services.
[0049] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A station service robot voice question-answering system based on passenger travel intention recognition, characterized in that: It includes a corpus construction module, a travel question and answer voice trigger word identification module, a passenger travel intention recognition module, a multi-round question and answer generation module, and a voice question and answer interface integrated with cloud computing; The corpus construction module is used to construct a speech corpus and a question-and-answer corpus. The speech corpus stores surveillance videos in the station, self-portrait videos of passengers, radio recordings of the station, and related interview documentaries. The speech corpus includes question-and-answer scenarios of route inquiries, ticket purchase instructions, and carriage conditions during passengers' station travel. The question-and-answer corpus is provided with conversation data about passengers' station travel obtained from the official website of the station and travel forums on social media platforms. The conversation data about station travel includes question-and-answer conversations between passengers and answers to passengers' questions by station staff. The travel question and answer voice trigger word discrimination module is equipped with a trained WaveNet model, which is used to separate multi-person audio and extract voice for recognition, and send the recognized voice to the passenger travel intention recognition module; The passenger travel intention recognition module converts speech into natural language segments, understands the contextual meaning of the natural language segments, and outputs the passenger travel intention recognition result; The travel question-and-answer voice trigger word identification module sends the recognized voice to the multi-round question-and-answer generation module to extract the voice feature vector, and transmits it to the voice dialogue model Mini-Omni to decode the voice feature vector, generate the corresponding text token and voice token, and perform fusion output; The voice question-and-answer interface integrated with cloud computing is used to display the output results of the multi-round question-and-answer generation module.
2. A station service robot voice question-answering system based on passenger travel intention recognition according to claim 1, characterized in that: The specific execution steps of the travel question and answer voice trigger word identification module are as follows: A1. After receiving a mixed audio signal containing multiple passengers’ voices, it is input into the trained WaveNet model to separate the individual audio data of each passenger; A2. Use Soundex method to convert speech into corresponding sound code; A3. Compare the converted audio code with the pre-stored codes in the speech corpus, calculate the Levenshtein distance between the query code and the code in the library, convert it into a similarity score, find the voice segment most similar to the passenger voiceprint information, and output it as the search result.
3. A station service robot voice question-answering system based on passenger travel intention recognition according to claim 1, characterized in that: The passenger travel intention recognition module uses the Transformer model to convert speech into natural language fragments and sends them to the RoBERTa model installed in the multi-round question and answer generation module to understand the contextual meaning.