Multi-language natural voice interaction method and system for coffee robot

Abstract

The invention relates to the technical field of voice services, in particular to a multi-language natural voice interaction method and system for a coffee robot, and the method comprises the steps: collecting a voice signal of a user, carrying out the recognition of the language type of the voice signal, obtaining the type information of the voice signal, activating a corresponding language recognition model according to the type information, and carrying out the recognition of the type information; performing semantic analysis processing on the voice signal, generating language text information corresponding to the voice signal, performing coffee customization demand analysis on a user based on the language text information to obtain coffee customization information of the user, initiating a voice interaction behavior to the user according to the coffee customization information to obtain an interaction feedback instruction of the user, and sending the interaction feedback instruction to the user. And a corresponding coffee beverage is produced according to the interaction feedback instruction and the coffee customization information.

Description

Technical Field

[0001] This invention relates to the field of voice service technology, and more specifically, to a multilingual natural voice interaction method and system for coffee robots. Background Technology

[0002] Modern consumers lead fast-paced lives and have high demands for convenience and efficiency in consumption. In the coffee consumption sector, consumers have their own taste preferences for coffee quality and personalized customization. Current coffee robots usually use smart touch screens for interaction, which is relatively inefficient and cannot provide services for consumers with multiple voices. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a multilingual natural voice interaction method and system for coffee robots, so as to achieve efficient provision of personalized coffee customization services.

[0004] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions: According to one aspect of the present invention, a multilingual natural voice interaction method for a coffee robot is provided, comprising: Collect the user's voice signal and identify the language type of the voice signal to obtain the voice signal type information; Activate the corresponding language recognition model based on the type information to perform semantic analysis on the speech signal and generate language text information corresponding to the speech signal; Based on the language and text information, the user's coffee customization needs are analyzed to obtain the user's coffee customization information; Based on the coffee customization information, a voice interaction is initiated with the user to obtain the user's interaction feedback instructions, and a corresponding coffee beverage is produced based on the interaction feedback instructions and the coffee customization information.

[0005] According to another aspect of the present invention, a multilingual natural voice interaction system for a coffee robot is provided, comprising: The speech recognition module is used to collect the user's speech signal and identify the language type of the speech signal to obtain the speech signal type information; The text interpretation module is used to activate the corresponding language recognition model according to the type information, so as to perform semantic analysis processing on the speech signal and generate language text information corresponding to the speech signal; The coffee customization module is used to parse the user's coffee customization needs based on the language text information to obtain the user's coffee customization information; The interactive feedback module is used to initiate voice interaction with the user based on the coffee customization information, to obtain the user's interactive feedback instructions, and to produce the corresponding coffee beverage based on the interactive feedback instructions and the coffee customization information.

[0006] As can be seen from the above technical solution, the multilingual natural voice interaction method for coffee robots provided by the present invention has the following beneficial effects: This invention can accurately identify multiple language types and adapt to different users in terms of language recognition. Semantic analysis generates text information to ensure accurate understanding of needs. It can analyze customized coffee needs in detail and obtain complete customized information. Feedback instructions can be obtained through voice interaction to improve customized content. Finally, the beverage is produced by combining information. This not only meets users' personalized needs and improves the consumption experience, but also improves service efficiency, reduces labor costs, and enhances the competitiveness of coffee robots in the market. Attached Figure Description

[0007] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort: Figure 1 This is a schematic diagram illustrating the steps of a multilingual natural voice interaction method for a coffee robot provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a multilingual natural voice interaction system for a coffee robot, provided in an embodiment of the present invention. Detailed Implementation

[0008] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0009] Modern consumers lead fast-paced lives and have high demands for convenience and efficiency in consumption. In the coffee consumption sector, consumers have their own taste preferences for coffee quality and personalized customization. Current coffee robots usually use smart touch screens for interaction, which is relatively inefficient and cannot provide services for consumers with multiple voices.

[0010] In view of this, the present invention provides a multilingual natural voice interaction method for coffee robots, the steps of which are as follows: Figure 1 As shown, it includes: The first step is to collect the user's voice signal and identify the language type of the voice signal to obtain the voice signal type information.

[0011] Specifically, in the first step of the embodiment provided by the present invention, the coffee robot is equipped with audio acquisition devices such as a microphone array. When the user approaches the robot and starts to speak, the microphone array captures the surrounding sound signals, converts them into electrical signals, and then processes them through analog-to-digital conversion to convert the analog signals into digital voice signals.

[0012] More specifically, the collected speech signals are analyzed using pre-trained basic language recognition models. These models are usually trained on a large amount of speech data from different languages, such as deep learning models like convolutional neural networks (CNN), recurrent neural networks (RNN) and their variants (LSTM, GRU). The models extract features from the speech signals, such as phonemes, tones, and spectrograms, and then determine the basic language type of the speech signal based on these features, such as English, Chinese, French, Japanese, etc.

[0013] More specifically, different languages ​​exhibit significant differences in phonetic features, grammatical structures, and vocabulary. Only by first identifying the basic language type can we lay the foundation for selecting a suitable language recognition model for more accurate semantic analysis. For example, English and Chinese have distinct phonetic characteristics; English has a rich combination of vowels and consonants, while Chinese has tones. By identifying the basic phonetic type, we can initially distinguish between them and adopt processing methods suitable for that language.

[0014] More specifically, a speech dataset containing various accents and dialects is constructed. These datasets cover speech samples from different regions and populations. After determining the basic language type, specific features of the speech signal are further extracted for that language, such as pronunciation habits, intonation patterns, and pronunciation of specific words. These features are then matched with features in the accent and dialect datasets. Machine learning algorithms (such as support vector machines and decision trees) or deep learning models (such as attention mechanism models) are used to determine the specific accent or dialect type of the speech signal. For example, for Chinese, it is determined to be Northeastern accent, Cantonese, Sichuan dialect, etc.; for English, it is determined to be American English, British English, Indian English, etc.

[0015] More specifically, even with the same basic language, different accents and dialects can have significant differences in pronunciation and vocabulary. For example, American English and British English differ in the pronunciation and spelling of some words, and Cantonese and Mandarin have obvious differences in vocabulary and pronunciation. Recognizing accents and dialects can improve the accuracy of speech recognition, avoid semantic misunderstandings caused by accents or dialects, and thus better meet the needs of different users.

[0016] More specifically, the basic language types and accent and dialect sub-types identified earlier are combined to form a complete speech signal type information. For example, "Chinese" and "Sichuan dialect" are combined into "Chinese-Sichuan dialect", and "English" and "American English" are combined into "English-American English".

[0017] More specifically, complete information on the type of speech signal is crucial for selecting appropriate language recognition models and semantic analysis strategies. Different base languages ​​and their corresponding accents and dialects require different processing methods and language models. By combining base types and refined types, more precise guidance can be provided for subsequent semantic analysis, ensuring accurate understanding of the user's speech intent and improving the interactive effect and service quality of the coffee robot.

[0018] The second step is to activate the corresponding language recognition model based on the type information to perform semantic analysis processing on the speech signal and generate language text information corresponding to the speech signal.

[0019] Specifically, in the second step of the embodiment provided by the present invention, the coffee robot system pre-stores a variety of language recognition models for different language types (including different base languages, accents and dialects). When the type information of the speech signal is obtained (such as "Chinese - Sichuan dialect", "English - American English", etc.), the system will select a matching language recognition model from the model library based on the type information. For example, if the speech signal type information is "Japanese - Tokyo accent", the system will activate a language recognition model specifically trained for the Japanese Tokyo accent.

[0020] More specifically, the activated speech recognition model performs feature extraction on the acquired speech signal. This involves converting the speech signal into time-frequency domain features, such as Mel frequency cepstral coefficients (MFCC) and linear predictive cepstral coefficients (LPCC). Taking MFCC as an example, the model divides the speech signal into frames, applies windows, calculates the power spectrum of each frame, converts it to the Mel frequency domain using a Mel filter bank, and finally performs discrete cosine transform to obtain MFCC features. These features can effectively represent the acoustic characteristics of the speech signal.

[0021] More specifically, speech recognition models utilize extracted features, combined with their internal acoustic and language models, to identify speech signals. The acoustic model describes the relationship between speech features and phonemes or syllables, while the language model considers the grammatical and semantic rules of the language to improve recognition accuracy. During the recognition process, the model continuously decodes and evaluates possible speech sequences, ultimately outputting the most likely text sequence. For example, for a Japanese speech sentence, the model uses the acoustic model to determine the phoneme corresponding to each pronunciation, and then combines these phonemes with the language model to form meaningful Japanese words and sentences. At the same time, the model also performs semantic analysis on the identified text sequence to understand the meaning of the sentence. This involves natural language processing techniques such as part-of-speech tagging, named entity recognition, and syntactic analysis to determine the specific intent and information expressed by the sentence.

[0022] More specifically, after speech recognition and semantic analysis, the system organizes the final text sequence and analysis results into language text information. This text information not only includes the text content corresponding to the speech signal, but also some additional annotation information, such as keywords and semantic categories, so as to further analyze user needs in the future.

[0023] More specifically, different languages, accents, and dialects have unique phonetic features and language rules. Using a language recognition model trained specifically for a particular type of information can better adapt to these characteristics, thereby improving the accuracy of speech recognition. For example, Sichuan dialect has some unique pronunciations and vocabulary. If a general Mandarin recognition model is used, there may be more recognition errors, while using a model trained specifically for Sichuan dialect can significantly improve the recognition effect.

[0024] More specifically, semantic analysis is a key step in understanding user needs. By performing semantic analysis on speech signals, speech can be converted into meaningful text information and the underlying intentions can be extracted. In the application scenario of coffee robots, accurately understanding the user's semantic information is crucial for interpreting customized coffee needs. For example, if a user says, "I want a latte with milk foam," through semantic analysis, the robot can accurately identify that the user needs a latte and has a personalized need for milk foam.

[0025] More specifically, the generated language text information is the foundation for subsequent steps such as coffee customization demand analysis and voice interaction. Only by accurately converting voice signals into text information can we conveniently process and analyze user needs, thereby achieving effective interaction with users and providing accurate services.

[0026] The third step is to analyze the user's coffee customization needs based on the language and text information to obtain the user's coffee customization information.

[0027] Specifically, in the third step of the embodiment provided by this invention, a keyword library containing various common coffee types is pre-established, such as latte, cappuccino, Americano, mocha, etc. A string matching algorithm is used to search for words in the keyword library in the language text information. For example, regular expression matching or a simple string search method is used. When the keyword "latte" is found in the text, it can be confirmed that the coffee type requested by the user is latte. If multiple coffee type keywords appear in the text, the final coffee type needs to be determined according to the context or specific rules. For example, if the text is "I'll consider latte first, or maybe cappuccino", the user's final choice of coffee type can be determined based on the word "or".

[0028] More specifically, a keyword library covering various personalized coffee customization needs is created, including keywords related to sugar content (e.g., less sugar, half sugar, more sugar), milk content (e.g., less milk, more milk), temperature (e.g., hot, iced), and additives (e.g., chocolate sauce, vanilla syrup). Similarly, a string matching algorithm is used to search for words in the customization keyword library within the text information. For example, when keywords such as "less sugar" or "with vanilla syrup" appear in the text, they are extracted as personalized customization information. This extracted personalized customization information is then associated with the previously determined coffee type. For instance, for the previously determined latte, if the customization needs of "less sugar" or "with vanilla syrup" are identified, it is clear that this is a personalized requirement for the latte.

[0029] More specifically, the confirmed coffee type information and the extracted personalized customization information are combined to form complete coffee customization information. This information can be stored in a structured data format, such as JSON or a dictionary. For example, the coffee type "latte" and the personalized customization information "less sugar, with vanilla syrup" can be integrated into a dictionary: {"coffee type": "latte", "personalization": "less sugar, with vanilla syrup"}.

[0030] More specifically, by identifying keywords for coffee type and personalized customization needs, user needs can be clearly broken down into two important parts, thus more accurately grasping the user's specific requirements. For coffee robots, identifying the coffee type is the foundation for making coffee, while understanding personalized customization needs can satisfy users' special taste preferences and provide users with coffee drinks that better suit their tastes.

[0031] More specifically, separating and extracting coffee type information and personalized customization information makes the subsequent processing flow clearer and more efficient. During subsequent voice interactions with users, targeted inquiries and confirmations can be made for different information. When generating coffee preparation instructions, this information can also be used to accurately control various parameters of coffee making, such as the amount of raw materials and the brewing temperature.

[0032] More specifically, this approach of breaking down requirements into different parts facilitates system expansion and upgrades. When new coffee types or personalized customization options emerge, only the corresponding keyword library needs to be updated to easily identify and process these new requirements without requiring large-scale modifications to the entire parsing process.

[0033] The fourth step involves initiating a voice interaction with the user based on the coffee customization information to obtain the user's interaction feedback instructions, and then producing the corresponding coffee beverage based on the interaction feedback instructions and the coffee customization information.

[0034] Specifically, in the fourth step of the embodiment provided by this invention, based on the coffee type in the coffee customization information, the corresponding coffee customization template is retrieved from the database. For example, if the user selects a latte, the latte customization template is retrieved. The template lists the customization elements that a latte usually includes, such as sugar content, milk foam amount, and coffee concentration. The coffee customization information provided by the user is compared with the customization template. By checking each element in the template one by one, it is determined which elements have been explicitly mentioned in the user's customization information and which have not been mentioned. For example, if the user only said they wanted a latte but did not mention the sugar content and milk foam amount, then the sugar content and milk foam amount are uncustomized elements. All uncustomized elements are collected to form an uncustomized element set. This set can be represented by a list or other suitable data structure for convenient subsequent processing.

[0035] More specifically, based on the set of uncustomized elements, corresponding query text is generated. The query text should be clear, concise, and conform to natural language habits so that users can easily understand it. For example, if the set of uncustomized elements includes sugar content and milk foam amount, the generated query text could be "What sugar content and how much milk foam do you want for this latte?" The coffee robot uses speech synthesis technology to convert the query text into a speech signal and plays it to the user through a speaker. At the same time, it turns on the microphone to receive the user's feedback speech. It uses the previously mentioned language recognition model to recognize the user's feedback speech, converts it into text information, and obtains interactive feedback instructions.

[0036] More specifically, the interactive feedback instructions are parsed to extract information related to the uncustomized elements. For example, if the user replies "less sugar, more milk foam", the sugar content is extracted as "less sugar" and the amount of milk foam is extracted as "more milk foam". The extracted information is then filled into the set of uncustomized elements to complete the customization of the uncustomized elements. This is how the supplementary customization information for coffee customization is obtained.

[0037] More specifically, the initial coffee customization information and supplementary customization information are integrated to form a complete and detailed coffee customization requirement. For example, the previous latte information and the supplementary "less sugar, more milk foam" information are combined. Based on the integrated information, instructions that the coffee preparation device can understand are generated. These instructions will clearly tell the coffee preparation device the amount of various raw materials, the brewing temperature, time and other parameters. For example, the instructions require the use of a specific amount of coffee powder, the addition of an appropriate amount of milk and frothing it into more milk foam, and the control of the amount of sugar added. The generated coffee preparation instructions are sent to the coffee preparation device, which drives the device to make coffee according to the instructions, and finally produces a coffee beverage that meets the user's needs.

[0038] More specifically, the coffee customization information initially provided by users may be incomplete. By analyzing the missing elements and interacting with the user, all of the user's customization requirements for coffee can be obtained, ensuring the accuracy of coffee preparation and meeting the user's personalized needs. For example, if the sugar content is not asked, the coffee may not meet the user's taste preferences.

[0039] More specifically, using voice interaction to communicate with users aligns with natural communication habits, making users feel more comfortable and at ease. Users can easily supplement their needs through relaxed communication without having to complete customization through complex operations or input.

[0040] More specifically, by combining complete customized information to generate accurate coffee preparation instructions, the coffee preparation device can ensure that coffee is made according to unified standards and the user's specific requirements, thereby improving the quality and consistency of the coffee. This can provide users with stable and high-quality coffee beverages and enhance user satisfaction with the coffee robot service.

[0041] As can be seen from the above technical solution, the multilingual natural voice interaction method for coffee robots provided by the present invention has the following beneficial effects: This invention can accurately identify multiple language types and adapt to different users in terms of language recognition. Semantic analysis generates text information to ensure accurate understanding of needs. It can analyze customized coffee needs in detail and obtain complete customized information. Feedback instructions can be obtained through voice interaction to improve customized content. Finally, the beverage is produced by combining information. This not only meets users' personalized needs and improves the consumption experience, but also improves service efficiency, reduces labor costs, and enhances the competitiveness of coffee robots in the market.

[0042] Furthermore, the steps of collecting the user's voice signal and identifying the language type of the voice signal to obtain the voice signal type information include: S11: Collect the user's voice signal and perform basic voice type recognition on the voice signal to obtain the basic type of the voice signal; S12: Based on the basic type, the speech signal is subjected to accent and dialect type recognition to obtain the refined type of the speech signal; S13: Combine the basic type with the refined type to obtain the type information of the speech signal.

[0043] Specifically, the coffee robot is equipped with a highly sensitive microphone array. When a user approaches the robot and begins to speak, the microphone array can capture surrounding sound signals from all directions. These sound signals exist in analog form and are first amplified by a preamplifier to enhance signal strength. Then, an analog-to-digital converter (ADC) converts the analog signal into a digital signal for subsequent computer processing. To ensure the quality of the acquired voice signal, the system performs preliminary filtering to remove environmental noise interference, such as filtering out high-frequency noise using a low-pass filter.

[0044] More specifically, recognition is performed using pre-trained multilingual basic speech recognition models. These models are trained on large-scale multilingual speech datasets and commonly use deep learning architectures, such as convolutional neural networks (CNNs) and variants of recurrent neural networks (RNNs), such as long short-term memory networks (LSTMs) or gated recurrent units (GRUs). The model first extracts features from the acquired digital speech signals, including time-domain features (such as zero-crossing rate and energy) and frequency-domain features (such as Mel-frequency cepstral coefficients, MFCCs). Then, these features are input into the model, which outputs the most likely basic language type of the speech signal, such as English, Chinese, or French, based on the degree of matching between the features and different language feature patterns.

[0045] More specifically, different basic languages ​​have huge differences in phonetics, grammar, and vocabulary. For example, English is a phonetic language with a rich combination of vowels and consonants, while Chinese is an ideographic language with tones. Only by first determining the type of basic language can we select a suitable language model for subsequent semantic analysis and interaction, thereby improving the accuracy and efficiency of recognition. Speech recognition models for different basic languages ​​are trained according to the characteristics of that language. Using specialized models can better adapt to the characteristics of the language, thereby improving the accuracy of recognition.

[0046] More specifically, after determining the basic language type, features related to accent and dialect are further extracted for that language. These features include prosodic features of pronunciation (such as intonation and speech rate variations), pronunciation patterns of specific words, and unique combinations of phonemes. For example, for Chinese, different dialects have significant differences in the pronunciation of tones and the use of vocabulary. By analyzing the frequency and duration variations of tones and the pronunciation of specific dialect words, dialect-related features can be extracted.

[0047] More specifically, accent and dialect recognition models specifically trained for this basic language are used. These models also employ machine learning or deep learning algorithms, learning the feature patterns of various accents and dialects by training on a large number of speech samples with different accents and dialects. The extracted accent and dialect features are then input into the model, which determines the specific accent or dialect type of the speech signal based on the feature matching, such as American English, British English, Sichuan dialect, Cantonese, etc.

[0048] More specifically, even within the same basic language, different accents and dialects can have significant differences in pronunciation, vocabulary, and grammar. For example, American English and British English differ in the pronunciation and spelling of some words, and Cantonese and Mandarin differ significantly in vocabulary and pronunciation. Recognizing accents and dialects can further improve the accuracy of speech recognition, avoid semantic misunderstandings caused by accents or dialects, and demonstrate respect and attention to the user's individuality, providing more thoughtful and accurate services and enhancing the user experience.

[0049] More specifically, the obtained basic language type and accent are combined with the dialect refinement type. For example, if the basic type is "Chinese" and the refinement type is "Sichuan dialect", then the combined speech signal type information is "Chinese-Sichuan dialect". This combined information can be stored in the system in text form, which makes it convenient to call the corresponding language recognition model for semantic analysis based on different language type information.

[0050] More specifically, combining basic and refined types can provide complete language information about the speech signal. This complete information helps the system to more accurately call appropriate language recognition models and semantic analysis strategies, ensuring accurate understanding and processing of the user's speech signal, thereby better meeting the user's customized coffee needs.

[0051] Furthermore, while generating the language text information corresponding to the speech signal, user tags are generated for the language text information based on the type information and audio characteristics of the speech signal, and timestamps are generated for the language text information based on the acquisition time of the speech signal. Language text information with timestamps within a specified time range and having the same user tags is aggregated and processed to serve as the overall language text information for the user.

[0052] Furthermore, if there are multiple users' language text information within a specified time range, the language text information of each user is compared to identify the language text information that prioritizes the beverage customization service, so as to execute the subsequent service process.

[0053] Specifically, based on the previously identified speech signal type information, such as "Chinese - Sichuan dialect" or "English - American English," this is used as part of the user tag. The audio characteristics of the speech signal are analyzed, including pitch, timbre, and speech rate. For example, the pitch is determined by calculating the fundamental frequency of the speech signal, the timbre is judged by spectrum analysis, and the speech rate is determined by counting the number of syllables per unit time. These audio characteristics are quantified and classified and also added to the user tag. For example, if a user's speech signal type is "Chinese - Sichuan dialect," with a high pitch and a fast speech rate, their user tag would be "Chinese - Sichuan dialect - high pitch - fast speech rate."

[0054] More specifically, when acquiring voice signals, the system records the current time, usually accurate to the millisecond level, and uses this time as the timestamp of the corresponding language text information. For example, if the voice signal was acquired at 10:30:15.234 on October 1, 2024, then the timestamp of the language text information would be "2024-10-01 10:30:15.234". A specified time range can be set, such as counting back 5 minutes from the current time. The system will then filter out all language text information whose timestamps fall within this specified time range. For the filtered language text information, it will be grouped according to user tags, and language text information with the same user tags will be grouped together and merged into a single piece of language text information. This text information can then be pieced together chronologically to form a complete dialogue record.

[0055] More specifically, different users' voice signals may differ in language type and audio characteristics. By generating user tags, different users can be accurately identified and distinguished. Even if multiple users interact with the coffee robot at the same time, the language text information of each user can be clearly distinguished, which facilitates the individual processing of each user's needs. The interaction between a user and the coffee robot may be carried out in multiple sessions, and the language text information of each interaction is incomplete when viewed alone. By generating timestamps and summarizing the language text information with the same user tags within a specified time range, the multiple interactions of the user can be integrated to form a complete dialogue, which helps to more comprehensively understand the user's needs and intentions.

[0056] More specifically, for the language text information of several users within a specified time range, the system will extract and analyze the content of each language text information, extract keywords related to beverage customization services, such as "coffee," "latte," and "customized," analyze the semantics of the language text information, and determine whether the user has clearly expressed a demand for beverage customization services. For example, "I want a latte, less sugar" clearly expresses a demand for coffee customization, while "Do you have coffee here?" is just an inquiry and has not yet clearly expressed a demand for customization services.

[0057] More specifically, based on the results of content analysis, it is determined whether each user prioritized requesting a customized beverage service. Priority is given to verbal and textual information that clearly expresses a need for customized beverages and is fully described. For example, if user A says, "I want a cappuccino with milk foam right now," and user B simply asks, "What coffee do you have?", then user A's verbal and textual information will be identified as prioritizing a request for customized beverage services.

[0058] More specifically, once the system confirms the user's text message requesting a customized beverage service, it will execute subsequent service processes based on that user's information, including parsing the coffee customization request, interacting with the user via voice, and preparing the coffee beverage.

[0059] More specifically, when multiple users interact with the coffee robot simultaneously, clearly defining the priority service recipients can improve service efficiency. Prioritizing users who have clearly stated their beverage customization needs can meet their requirements more quickly, avoiding service process chaos and inefficiency caused by handling multiple users' different needs at the same time. For users who urgently need beverage customization services, timely response and processing can improve their satisfaction. By identifying priority service recipients, it can be ensured that these users' needs are met in a timely manner, thereby improving the overall user experience.

[0060] Furthermore, the step of parsing the user's coffee customization request based on the aforementioned language text information to obtain the user's coffee customization information includes: S31: Based on the language text information, perform keyword recognition of coffee type to confirm the user's coffee type information; S32: Based on the language text information, perform keyword recognition on the personalized customization requirements of coffee type to obtain the user's personalized customization information for the coffee type. S33: Combine the coffee type information with the personalized customization information to obtain the user's coffee customization information.

[0061] Specifically, compile a list of common coffee types beforehand, such as latte, cappuccino, Americano, mocha, espresso, macchiato, etc., to build a comprehensive keyword library. These keywords can be stored using data structures such as lists and dictionaries.

[0062] More specifically, string matching algorithms are used to scan the language text information word by word to find whether there is a coffee type name in the keyword database. For example, if exact matching is used, if "latte" appears in the text, it is determined that the coffee type requested by the user is latte. Fuzzy matching technology can also be used to deal with the situation where the user's expression is not precise. For example, when recognizing "latte", it can also be associated with "latte".

[0063] More specifically, if multiple coffee type keywords appear in the language text information, it is necessary to make a judgment based on the contextual semantics. For example, if the text is "I want to try a latte first, but a cappuccino seems good too", the user's preferred coffee type can be determined based on the expression "try first".

[0064] More specifically, coffee type is the core of a user's coffee customization needs. Determining the coffee type first provides a basic framework for subsequent personalized customization and clarifies the basic direction of coffee making. If the coffee type cannot be accurately identified, the subsequent customization service will lose its basis. Identifying the coffee type separately facilitates the identification and matching of corresponding personalized customization information for that type, making the entire demand analysis process clearer and more orderly.

[0065] More specifically, various keywords related to coffee customization are collected and categorized according to different dimensions, such as sugar content (less sugar, half sugar, more sugar, no sugar), milk content (less milk, more milk, normal milk content), temperature (hot, iced, room temperature), and additives (chocolate sauce, vanilla syrup, caramel, cinnamon powder), etc. The language text information is then scanned in detail to identify keywords related to personalized customization needs. Tools such as regular expressions can be used to extract these keywords more efficiently. For example, sugar content information can be extracted by matching the pattern "[sugar content related words]". The extracted personalized customization keywords are then associated with the previously confirmed coffee type information. For example, if the coffee type has been determined to be latte, and "less sugar" and "with vanilla syrup" are identified, then these customization needs are clearly for latte coffee.

[0066] More specifically, different users have different taste preferences for coffee. By identifying personalized customization needs, we can meet users' special needs, provide them with coffee drinks that better suit their tastes, and improve user satisfaction. Clear personalized customization information is the key basis for coffee robots to accurately make coffee. For example, knowing the sugar content and additive information is essential to accurately control the amount of raw materials used and make coffee with the right flavor.

[0067] More specifically, an appropriate data structure should be used to integrate coffee type information and personalized customization information. This can be done using a dictionary, such as {"Coffee Type": "Latte", "Personalization": "Less Sugar, Vanilla Syrup"}; or by creating an object class that contains the coffee type and customization requirements. After integration, check the completeness of the customization information. If any important customization elements are missing, further clarification can be sought during subsequent interactions with the user.

[0068] More specifically, coffee type information and personalized customization information are both important components of users' coffee customization needs. Only by combining them can a complete and specific customization information be formed, which provides accurate guidance for subsequent coffee making and service processes. The integrated coffee customization information is presented in a unified data structure, which facilitates the system's storage, transmission and processing, and also facilitates docking with coffee making equipment to achieve precise coffee making.

[0069] Furthermore, the steps of initiating voice interaction with the user based on the coffee customization information to obtain the user's interaction feedback instructions, and producing the corresponding coffee beverage based on the interaction feedback instructions and the coffee customization information include: S41: Parse the non-customized elements of the coffee customization information to obtain the set of non-customized elements of the user's coffee customization information; S42: Generate an inquiry text based on the set of uncustomized elements, and initiate a voice interaction with the user based on the inquiry text to receive the user's feedback voice and recognize the feedback voice to obtain an interaction feedback instruction; S43: Customize the uncustomized element set according to the interactive feedback instruction to obtain supplementary customization information for the coffee customization information; S44: Combine the coffee customization information with the supplementary customization information to generate a coffee preparation instruction, thereby driving the coffee preparation device to produce the corresponding coffee beverage.

[0070] Specifically, the system pre-stores complete customized templates corresponding to different types of coffee. Each template contains all the customized elements involved in that coffee. For example, the template for latte coffee includes elements such as coffee concentration, milk foam amount, sugar content, and temperature.

[0071] More specifically, based on the coffee type in the user's coffee customization information, the corresponding customization template is accurately retrieved from the database. The user's coffee customization information is then compared element-by-element with the retrieved template. For example, if the user's customization information only mentions wanting a latte without specifying sugar content, temperature, etc., comparing the template reveals these missing elements. All elements that do not match the specific customization content are then compiled into a set of uncustomized elements. These uncustomized elements can be stored using a list or set data structure, such as {"sugar content", "temperature"}.

[0072] More specifically, the coffee customization information initially provided by the user may not be complete. By comparing it with the template, all the undefined customization elements can be identified, thereby ensuring that the complete customization requirements are obtained and avoiding the coffee not meeting the user's expectations due to missing information. Identifying the set of uncustomized elements is the basis for making effective inquiries to the user later. Only by clearly knowing which elements are not customized can we accurately ask the user for relevant information.

[0073] More specifically, based on the set of uncustomized elements, and combined with natural language expression habits, query text is generated. For example, if the uncustomized elements are sugar content and temperature, the generated query could be "What sugar content would you like for this latte? Would you prefer it hot or cold?" To make the query more friendly and natural, some polite phrases such as "Hello" or "Excuse me" can be added to the text.

[0074] More specifically, the generated inquiry text is converted into a speech signal using speech synthesis technology. Speech synthesis typically employs a pre-trained speech synthesis model to generate natural and fluent speech based on the text content. This speech is then played through the coffee robot's speaker to initiate an interactive inquiry to the user. Simultaneously, the microphone is activated to prepare for receiving the user's feedback speech.

[0075] More specifically, the user's feedback speech is processed using the language recognition model adapted to the user's speech type mentioned earlier. This model performs operations such as feature extraction, acoustic decoding, and language model matching on the speech signal to convert the speech into text form, thereby obtaining interactive feedback instructions.

[0076] More specifically, by asking about non-customized elements through voice interaction, the system can directly communicate with users to obtain their specific requirements for these elements, ensuring that the final customized information is what the user truly wants. The voice interaction method aligns with people's daily communication habits, allowing users to supplement customized information in a relaxed and natural atmosphere, thus enhancing the user's experience of interacting with the coffee robot.

[0077] More specifically, the interactive feedback commands are analyzed in depth to extract key information related to the uncustomized elements. For example, if a user replies "less sugar, hot", the sugar content is parsed as "less sugar" and the temperature as "hot". For complex replies, semantic understanding technology in natural language processing is needed to accurately determine the user's intent.

[0078] More specifically, the parsed key information is matched with the elements in the set of uncustomized elements to complete the customization of the uncustomized elements. For example, for the set of uncustomized elements {"sweetness", "temperature"}, "less sugar" and "hot" are filled into the corresponding elements to form supplementary customized information. The format can be consistent with the original coffee customization information, such as {"sweetness": "less sugar", "temperature": "hot"}.

[0079] More specifically, customizing uncustomized elements based on user feedback can improve coffee customization information, making the entire customization information more complete and accurate, providing a more detailed basis for precise coffee making. Matching feedback instructions with uncustomized elements ensures the consistency and relevance of the supplemented customization information with the original customization information, facilitating subsequent unified processing.

[0080] More specifically, the original coffee customization information and supplementary customization information are combined to form a complete and detailed final coffee customization requirement. For example, the original coffee customization information is {"Coffee Type": "Latte"}, and the supplementary customization information is {"Sweetness": "Less Sugar", "Temperature": "Hot"}. After integration, we get {"Coffee Type": "Latte", "Sweetness": "Less Sugar", "Temperature": "Hot"}.

[0081] More specifically, based on the integrated information, instructions suitable for the coffee preparation device to understand and execute are generated. These instructions precisely specify various parameters in the coffee making process, such as the amount of coffee powder, milk, sugar, heating temperature, and time. The format of the instructions is usually matched with the interface protocol of the coffee preparation device to ensure that the device can accurately receive and execute them. The generated coffee preparation instructions are sent to the coffee preparation device. After receiving the instructions, the device operates according to the parameters in the instructions to complete the coffee making and ultimately provide users with coffee beverages that meet their customized requirements.

[0082] More specifically, the coffee preparation instructions generated by combining complete customized information can precisely control every step and parameter of coffee making, ensuring that the coffee preparation device produces coffee drinks that meet the taste and quality standards according to the user's requirements. From obtaining user customized information to generating preparation instructions and driving the device to make coffee, a complete service process loop is formed, realizing the entire process of coffee robot from interaction to service, and providing users with a one-stop coffee customization service.

[0083] Furthermore, the step of parsing the non-customized elements of the coffee customization information to obtain the set of non-customized elements of the user's coffee customization information includes: S411: Retrieve the corresponding coffee customization template based on the coffee type in the coffee customization information; S412: Based on the coffee customization template, perform information matching of coffee customization elements on the coffee customization information to obtain the user's customized and non-customized elements for coffee; S413: Integrate the aforementioned uncustomized elements to obtain a set of uncustomized elements for the user's coffee customization information.

[0084] Specifically, the system pre-builds a database containing various coffee type customization templates. Upon receiving a user's coffee customization information, it extracts the key information of the coffee type. For example, if the user's coffee customization information indicates a "cappuccino," the system will search the database using "cappuccino" as the keyword.

[0085] More specifically, each template in the database corresponds to a specific type of coffee. By searching, you can find a customized template that corresponds to "cappuccino". This template details all the customized elements involved in making a cappuccino, such as the fineness of the milk foam, the ratio of coffee to milk, the sugar content, and the temperature.

[0086] More specifically, different types of coffee have their unique preparation requirements and customization options. Using corresponding coffee customization templates as standardized references ensures that the system fully considers all customization elements of that coffee type, avoiding the omission of important information. Pre-stored templates can be quickly retrieved, reducing the time and workload of re-organizing elements each time user customization information is processed, and improving the system's processing efficiency.

[0087] More specifically, all elements in the coffee customization template are listed one by one to form a complete element list. For example, the element list for cappuccino includes "milk foam fineness," "coffee-to-milk ratio," "sweetness," "temperature," and "whether to add chocolate powder," etc. The coffee customization information provided by the user is compared with the element list. For each element, it is checked whether it is explicitly mentioned in the user's customization information. If so, the element is marked as a customized element; otherwise, it is marked as a non-customized element. For example, if the user's customization information only mentions "cappuccino, less sugar," then "sweetness" is a customized element, while "milk foam fineness," "coffee-to-milk ratio," "temperature," and "whether to add chocolate powder" are non-customized elements.

[0088] More specifically, by matching users' customized information with template elements, the system can accurately identify users' explicit customization needs and those not yet mentioned. This helps the system interact with users in a targeted manner, asking only about truly uncertain elements, avoiding unnecessary inquiries, improving interaction efficiency, and clearly identifying customized and non-customized elements. This ensures that all necessary information can be obtained during the subsequent coffee-making process, thereby producing coffee drinks that meet users' expectations.

[0089] More specifically, elements marked as uncustomized can be collected and stored using data structures such as lists and sets. For example, in the cappuccino example above, uncustomized elements such as "milk foam fineness," "coffee-to-milk ratio," "temperature," and "whether to add chocolate powder" can be collected into a list. The collected uncustomized elements can be organized into a set, which clearly shows the elements in the user's coffee customization information that are not yet clear, making it easier to ask the user for this information later.

[0090] More specifically, by integrating the uncustomized elements into a set, the system can manage and process these elements in a unified manner. When interacting with users via voice in the future, clear and organized query text can be generated based on the elements in this set, ensuring that the system can accurately obtain the user's specific requirements for these uncustomized elements. The set of uncustomized elements presents the missing parts of the user's customized information in an intuitive way, enabling the system and operators to quickly understand what needs to be further communicated with the user, and providing a clear direction for the subsequent service process.

[0091] Furthermore, it also includes: when the content fed back by the user's voice signal contains the user's account, retrieving the historical customization record corresponding to the user's account from the database, and analyzing the user's preference information based on the historical customization record, so as to generate an inquiry text based on the preference information and the set of non-customized elements.

[0092] Specifically, after semantic analysis of the user's voice signal to generate language text information, the system will perform keyword matching on the text information to check whether it contains keywords related to the user's account, such as account number, username, etc. For example, if the user says "I am the user of account 12345", the system will recognize "12345" as the user account. Once the user account information is identified, the system will accurately extract it so that the relevant records can be retrieved from the database later.

[0093] More specifically, by analyzing users' historical customization records and preference information, more personalized services can be provided. Question texts can be generated based on user preferences, making users feel that the coffee robot pays attention to them and understands them, thereby enhancing user satisfaction and loyalty. If user needs can be predicted in advance based on user preferences, inquiries about non-customized elements can be more targeted, reducing unnecessary inquiries, improving communication efficiency, and saving users time and energy.

[0094] More specifically, the system uses the extracted user accounts as query criteria to search the database, which stores each user's historical customization records. These records contain detailed information about the user's past coffee customizations, such as coffee type, personalized customization requirements (sweetness, milk foam amount, additives, etc.), and customization time. The system filters out all historical customization records corresponding to the user account from the database and loads these records into the system for further analysis.

[0095] More specifically, combining user preferences and non-customized elements to generate query text helps to more accurately obtain users' customization needs. For example, when a user has a clear preference for a certain element, mentioning that preference in the query can guide the user to confirm or adjust their needs, thereby ensuring that the final coffee made is more in line with the user's taste. Although users have certain preferences, their needs may also change. By asking questions, the user's historical preferences are taken into account, and the user is given the opportunity to express their current different needs, making coffee customization more flexible and accurate.

[0096] More specifically, the system performs statistical analysis on the retrieved historical customization records. For example, it analyzes the most frequently chosen coffee type and calculates the user's average preference for personalized customization elements such as sugar content and milk foam amount. If a user has consistently chosen a low-sugar latte in past customizations, it can be inferred that the user prefers low-sugar lattes. In addition to simple statistical analysis, the system also identifies patterns in user customization behavior. For instance, a user typically chooses an Americano in weekday mornings and prefers a cappuccino in weekend afternoons, reflecting the user's coffee preference patterns at different times.

[0097] More specifically, the analyzed user preference information is integrated with the set of uncustomized elements. For example, if a user prefers a less sugar latte, but the sugar content is not mentioned in the customized information, then when considering the uncustomized element "sugar content," the user's preference for less sugar will be taken into account. Based on the integrated information, a query text that conforms to natural language expression habits will be generated. The query text must consider both the user's preference and the uncustomized elements. For example, for the case of the aforementioned preference for less sugar without mentioning sugar content, the query text could be, "You usually like less sugar in your coffee, would you like less sugar in this latte as well?" More specifically, by making full use of the user's historical customization records stored in the database and transforming this data into valuable information to guide the interaction process between the coffee robot and the user, this data-driven decision-making approach can improve the system's intelligence level and service quality.

[0098] Based on the technical content of the multilingual natural voice interaction method for coffee robots described in the above-disclosed embodiments, the present invention provides a multilingual natural voice interaction system for coffee robots, the structure of which is as follows: Figure 2 A method for implementing a multilingual natural speech interaction method for a coffee robot as described in any one of the first aspects, comprising: The speech recognition module is used to collect the user's speech signal and identify the language type of the speech signal to obtain the speech signal type information; The text interpretation module is used to activate the corresponding language recognition model according to the type information, so as to perform semantic analysis processing on the speech signal and generate language text information corresponding to the speech signal; The coffee customization module is used to parse the user's coffee customization needs based on the language text information to obtain the user's coffee customization information; The interactive feedback module is used to initiate voice interaction with the user based on the coffee customization information, to obtain the user's interactive feedback instructions, and to produce the corresponding coffee beverage based on the interactive feedback instructions and the coffee customization information.

[0099] In this embodiment, the specific implementation of each module in the above system embodiment is described in the above method embodiment, and will not be repeated here.

[0100] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0101] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0102] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-lingual natural voice interaction method for a coffee robot, characterized in that, The method comprises the following steps: Collecting a user's voice signal and identifying the language type of the voice signal to obtain type information of the voice signal; According to the type information, activating the corresponding language recognition model to perform semantic analysis processing on the voice signal to generate language text information corresponding to the voice signal; Based on the language text information, analyzing the user's coffee customization needs to obtain the user's coffee customization information; According to the coffee customization information, initiating a voice interaction behavior to the user to obtain the user's interaction feedback instruction, and according to the interaction feedback instruction and the coffee customization information, outputting the corresponding coffee beverage.

2. A multi-lingual natural speech interaction method for a coffee robot as claimed in claim 1, wherein, The step of collecting a user's voice signal and identifying the language type of the voice signal to obtain type information of the voice signal comprises: Collecting a user's voice signal and identifying the basic voice type of the voice signal to obtain the basic type of the voice signal; Based on the basic type, identifying the type of accent and dialect of the voice signal to obtain the refined type of the voice signal; Combining the basic type and the refined type to obtain the type information of the voice signal.

3. A multi-lingual natural speech interaction method for a coffee robot as claimed in claim 1, wherein, While generating the language text information corresponding to the voice signal, generating a user tag for the language text information according to the type information and the audio characteristics of the voice signal, and generating a timestamp for the language text information according to the collection time of the voice signal, and processing the language text information with the same user tag within the specified time range as the language text information of the user as a whole.

4. A multi-lingual natural speech interaction method for a coffee robot as claimed in claim 3, wherein, If there are several users' language text information within the specified time range, compare the content of each user's language text information to confirm the language text information that prioritizes the beverage customization service to execute the subsequent service process.

5. A multi-lingual natural speech interaction method for a coffee robot as claimed in claim 1, wherein, The step of analyzing the user's coffee customization needs based on the language text information to obtain the user's coffee customization information comprises: Based on the language text information, identifying the keywords of the coffee type to confirm the user's coffee type information; Based on the language text information, identifying the keywords of the individual customization needs of the coffee type to obtain the individual customization information of the user for the coffee type information; Combining the coffee type information and the individual customization information to obtain the user's coffee customization information.

6. A multi-lingual natural speech interaction method for a coffee robot as claimed in claim 1, wherein, The step of initiating a voice interaction behavior to the user according to the coffee customization information to obtain the user's interaction feedback instruction, and according to the interaction feedback instruction and the coffee customization information, outputting the corresponding coffee beverage comprises: Analyzing the undetermined elements of the coffee customization information to obtain an undetermined element set of the user's coffee customization information; Based on the undetermined element set, generating a query text, and according to the query text, initiating a voice interaction behavior to the user to receive the user's feedback voice and identify the feedback voice to obtain the interaction feedback instruction; According to the interaction feedback instruction, customizing the elements of the undetermined element set to obtain the supplementary customization information of the coffee customization information; The coffee customization information is combined with the supplementary customization information to generate coffee preparation instructions to drive a coffee preparation device to produce a corresponding coffee beverage.

7. A multi-lingual natural speech interaction method for a coffee robot as claimed in claim 6, characterized by, The step of analyzing uncustomized elements of the coffee customization information to obtain a set of uncustomized elements of the coffee customization information of the user includes: According to the coffee type in the coffee customization information, a corresponding coffee customization template is called; Based on the coffee customization template, the coffee customization information is matched for coffee customization elements to obtain customized elements and uncustomized elements of coffee of the user; Each of the uncustomized elements is integrated to obtain a set of uncustomized elements of the coffee customization information of the user.

8. A multi-lingual natural speech interaction method for a coffee robot as claimed in claim 6, characterized by, Also includes: When the content fed back by the voice signal of the user contains a user account, the historical customization record corresponding to the user account is called from the database, and the preference information of the user is analyzed according to the historical customization record, to generate inquiry text based on the preference information and the set of uncustomized elements.

9. A multi-lingual natural speech interaction system for a coffee robot, characterized by, Includes: A voice recognition module is configured to collect a voice signal of a user and recognize a language type of the voice signal to obtain type information of the voice signal; A text interpretation module is configured to activate a corresponding language recognition model according to the type information to perform semantic analysis on the voice signal and generate language text information corresponding to the voice signal; A coffee customization module is configured to analyze coffee customization needs of the user based on the language text information to obtain coffee customization information of the user; An interactive feedback module is configured to initiate a voice interaction with the user according to the coffee customization information to obtain an interactive feedback instruction of the user, and to produce a corresponding coffee beverage according to the interactive feedback instruction and the coffee customization information.

Images