Hotel intelligent voice dialogue method based on multi-modal interaction

Abstract

The invention relates to the technical field of information interaction. The hotel intelligent voice dialogue method based on multi-modal interaction comprises the steps that multi-modal data of a user in hotel voice dialogue are acquired, and the multi-modal data comprise voice data, facial expression video data and text interaction data; acquiring user acoustic features according to the voice data, and acquiring voice emotion features according to the user acoustic features; obtaining user visual features according to the facial expression video data, and obtaining visual emotion features according to the user visual features; and obtaining user text features according to the text interaction data, and obtaining text emotion features according to the user text features. According to the method, voice, facial expression videos and text interaction data are fused, so that the recognition precision of the emotional state and the demand intention of the user is improved, and the problem of misjudgment caused by information loss of a traditional single-mode system is solved.

Description

Technical Field

[0001] The present application relates to the field of information interaction technology, and in particular to a hotel intelligent voice dialogue method based on multimodal interaction. Background Art

[0002] With the development of intelligent voice interaction technology, intelligent dialogue systems in hotel scenarios have gradually become an important means to improve service efficiency.

[0003] Existing technologies have the following shortcomings: First, single-modal data cannot handle the correlation analysis of multi-dimensional information. For example, the contradiction between voice content and facial expressions can easily lead to a break in the interaction logic; and the feedback generation mechanism generally lacks emotional adaptability. Traditional systems mostly use a fixed tone and a single feedback form, which does not dynamically adjust according to the user's emotional state, resulting in a stiff interaction experience. Summary of the Invention

[0004] This application provides a hotel intelligent voice dialogue method based on multimodal interaction to solve the problem that single-modal data cannot handle the correlation analysis of multi-dimensional information, such as the contradiction between voice content and facial expressions, which easily leads to the problem of interaction logic fault.

[0005] In the first aspect, the present application provides a hotel intelligent voice dialogue method based on multimodal interaction, including: Acquire multimodal data of the user's voice conversation in the hotel, wherein the multimodal data includes voice data, facial expression video data, and text interaction data; Acquiring user acoustic features based on the voice data, and acquiring voice emotion features based on the user acoustic features; Acquiring user visual features based on the facial expression video data, and acquiring visual emotion features based on the user visual features; Acquire user text features based on the text interaction data, and acquire text sentiment features based on the user text features; The speech emotion feature, visual emotion feature, and text emotion feature are integrated to obtain a multimodal feature, and the user's comprehensive emotion state information and demand intention information are obtained based on the multimodal feature; Acquire hotel service information related to user needs based on the demand intention information; Obtaining user feedback information based on the comprehensive emotional state information and hotel service information, wherein the feedback information includes voice response, text response, and visual feedback information; The voice reply, text reply and visual feedback information are synchronously output to the user and updated according to the user's subsequent feedback dialogue status.

[0006] Optionally, the step of acquiring user acoustic features according to the voice data, and acquiring voice emotion features according to the user acoustic features includes: Preprocessing the voice data to obtain a preprocessed voice signal; Performing frame processing on the preprocessed speech signal to obtain a plurality of speech frames; Performing a short-time Fourier transform on each of the speech frames to obtain a spectrogram corresponding to each of the speech frames; Acquiring user acoustic feature data according to each of the spectrograms, wherein the user acoustic feature data includes Mel-frequency cepstral coefficients, acoustic fundamental frequency feature data, and acoustic short-time energy feature data; Inputting the Mel-frequency cepstral coefficients, acoustic fundamental frequency feature data and acoustic short-time energy feature data into a preset emotion recognition rule to obtain a comprehensive speech emotion feature; Acquiring emotion preset recognition rules according to the comprehensive speech emotion features; The user's emotional state information is acquired according to the preset emotion recognition rule, where the user's emotional state information includes positive state information, negative state information, and neutral state information.

[0007] Optionally, the step of obtaining user visual features based on the facial expression video data, and obtaining visual emotion features based on the user visual features, includes: Acquire user facial key point positioning information based on the facial expression video data, wherein the user facial key point positioning information includes an eye area image, a mouth area image, and a facial muscle area image; Normalizing the user's facial key point positioning information, adjusting the image size and brightness contrast, and obtaining processed user's facial key point positioning information; Extract features from the user's facial key point positioning information to obtain facial expression feature information and visual feature information; The facial expression feature information and visual feature information are matched with a preset facial expression emotion template to obtain visual emotion features.

[0008] Optionally, the step of obtaining user text features based on the text interaction data, and obtaining text sentiment features based on the user text features, includes: Performing word segmentation processing on the text interaction data to obtain a word segmentation text sequence; Convert the text sequence into a word vector matrix to obtain user text features; Performing semantic analysis on the user text features to obtain contextual semantic features; The text sentiment feature is obtained according to the context semantic feature.

[0009] Optionally, the step of fusing the voice emotion feature, the visual emotion feature, and the text emotion feature to obtain a multimodal feature, and obtaining the user's comprehensive emotion state information and demand intention information based on the multimodal feature includes: Align the dimensions of the speech emotion features, visual emotion features, and text emotion features, The aligned speech emotion features, visual emotion features, and text emotion features are fused to obtain multimodal features; Obtaining comprehensive status information of the user according to the multimodal features, Acquiring user emotion text features and intention text features based on the multimodal features; Acquiring comprehensive emotional state information based on the user's emotional text features; The user demand intention information is obtained according to the intention text features, and the demand intention information includes hotel reservation, service consultation, and complaint suggestions.

[0010] Optionally, the step of obtaining hotel service information related to user needs according to the demand intention information includes: Acquiring a user demand type according to the demand intention information, wherein the demand type includes room reservation, catering service, and facility inquiry; Acquire query hotel service data according to the demand type, and retrieve corresponding hotel service information according to the query hotel service data, wherein the hotel service database includes room type, price, restaurant menu, and facility opening hours data.

[0011] Optionally, the step of obtaining user feedback information based on the comprehensive emotional state information and hotel service information includes: Obtaining preset response tone information according to the comprehensive emotional state; Generating structured reply content according to the hotel service information, wherein the structured reply content includes text description, image link and speech synthesis text; Integrating the preset reply tone information and the structured reply content to generate voice reply, text reply and visual feedback information; User feedback information is obtained based on the voice response, text response and visual feedback information.

[0012] Secondly, this application also provides a hotel intelligent voice dialogue system based on multimodal interaction, including: A data acquisition module is used to obtain multimodal data of users' voice conversations in the hotel, wherein the multimodal data includes voice data, facial expression video data and text interaction data; A speech emotion feature extraction module, configured to obtain user acoustic features based on the speech data, and to obtain speech emotion features based on the user acoustic features; A visual emotion feature extraction module, configured to obtain user visual features based on the facial expression video data, and to obtain visual emotion features based on the user visual features; A text sentiment feature extraction module, configured to obtain user text features based on the text interaction data, and to obtain text sentiment features based on the user text features; An analysis module is used to fuse the voice emotion features, visual emotion features, and text emotion features to obtain multimodal features, and obtain the user's comprehensive emotional state information and demand intention information based on the multimodal features; An information retrieval module is used to obtain hotel service information related to user needs based on the demand intention information; a multimodal feedback generation module, configured to obtain user feedback information based on the comprehensive emotional state information and hotel service information, wherein the feedback information includes voice response, text response, and visual feedback information; The interactive output and dialogue state management module is used to synchronously output the voice reply, text reply and visual feedback information to the user, and update the dialogue state according to the user's subsequent feedback.

[0013] Optionally, the speech emotion feature extraction module includes: A speech signal preprocessing unit, configured to preprocess the speech data to obtain a preprocessed speech signal; A voice signal framing processing unit, configured to perform framing processing on the pre-processed voice signal to obtain a plurality of voice frames; a transform processing unit, configured to perform a short-time Fourier transform on each of the speech frames to obtain a spectrogram corresponding to each of the speech frames; an acoustic feature extraction unit, configured to obtain user acoustic feature data according to each of the spectrograms, wherein the user acoustic feature data includes Mel-frequency cepstral coefficients, acoustic fundamental frequency feature data, and acoustic short-time energy feature data; An emotion recognition rule application unit is used to input the Mel-frequency cepstral coefficients, acoustic fundamental frequency feature data and acoustic short-time energy feature data into a preset emotion recognition rule to obtain a comprehensive speech emotion feature; An emotion preset rule acquisition unit, configured to acquire an emotion preset recognition rule based on the comprehensive speech emotion feature; The state determination unit is used to obtain user emotional state information according to the preset emotion recognition rule, and the user emotional state information includes positive state information, negative state information and neutral state information.

[0014] In a second aspect, the present application provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, the hotel intelligent voice dialogue method based on multimodal interaction provided in the first aspect above is implemented.

[0015] Compared with related technologies, the hotel intelligent voice dialogue method based on multimodal interaction provided by this application has at least the following technical effects: The present invention realizes the hotel intelligent voice dialogue system through multimodal interaction technology, which has significant beneficial effects. By integrating voice, facial expression video and text interaction data, the recognition accuracy of the user's emotional state and demand intention is improved, and the misjudgment problem caused by the lack of information in the traditional single-modal system is solved. The dynamic weighted fusion strategy adapts to different interaction scenarios, combines hotel domain knowledge to optimize retrieval and feedback, and makes service information matching more accurate. The emotion-driven multimodal feedback generation mechanism customizes the tone and feedback form according to the user's comprehensive emotional state, enhancing the naturalness of interaction and user experience. The system improves data quality through preprocessing, framing, and feature extraction steps, combines lightweight algorithms to reduce computing power requirements, and adapts to edge computing devices in hotel scenarios.

[0016] The details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 Schematic diagram of a method flow according to an embodiment of the present invention.

[0018] Figure 2 FIG. 1 is a schematic diagram of a system structure according to an embodiment of the present invention.

[0019] Figure 3 This is a schematic diagram of the internal structure of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0020] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is described and illustrated below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely used to explain this application and are not intended to limit this application. Based on the embodiments provided in this application, all other embodiments obtained by those of ordinary skill in the art without making any creative efforts are within the scope of protection of this application.

[0021] Obviously, the drawings described below are merely examples or embodiments of the present application. Those skilled in the art can apply the present application to other similar scenarios based on these drawings without inventive effort. Furthermore, it is also understood that, although the effort involved in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, some design, manufacturing, or production changes based on the technical content disclosed in this application are merely conventional technical means and should not be construed as an insufficiency of the content disclosed in this application.

[0022] References to "embodiments" in this application mean that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it refer to independent or alternative embodiments that are mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described in this application may be combined with other embodiments unless there is a conflict.

[0023] Unless otherwise defined, technical or scientific terms used herein shall have the ordinary meaning as understood by persons of ordinary skill in the art to which this application belongs. The terms "a," "an," "an," "the," and similar expressions used herein do not denote quantitative limitations and may refer to either the singular or the plural. The terms "comprise," "include," "have," and any variations thereof, used herein, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or modules (units) is not limited to the listed steps or units but may also include steps or units not listed, or may include other steps or units inherent to the process, method, product, or apparatus. The terms "connected," "connected," "coupled," and similar expressions used herein are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. As used herein, "plurality" means two or more. "And / or" describes an association between associated objects, indicating that three possible relationships exist. For example, "A and / or B" may mean: A exists alone; A and B exist simultaneously; or B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific order for the objects. Example 1

[0024] The embodiment of the present invention provides a hotel intelligent voice dialogue method based on multimodal interaction. Figure 1 FIG. 1 is a flow chart according to an exemplary embodiment. Figure 1Shown, including: S1. Acquire multimodal data of a user's voice conversation in a hotel, wherein the multimodal data includes voice data, facial expression video data, and text interaction data; S2. Acquire user acoustic features according to the voice data, and acquire voice emotion features according to the user acoustic features; S3, obtaining user visual features based on the facial expression video data, and obtaining visual emotion features based on the user visual features; S4. Acquire user text features based on the text interaction data, and acquire text sentiment features based on the user text features; S5. Fusing the voice emotion feature, visual emotion feature, and text emotion feature to obtain a multimodal feature, and obtaining the user's comprehensive emotion state information and demand intention information based on the multimodal feature; S6. Acquire hotel service information related to the user's needs based on the demand intention information; S7. Obtaining user feedback information based on the comprehensive emotional state information and hotel service information, wherein the feedback information includes voice response, text response, and visual feedback information; S8. Synchronously output the voice reply, text reply, and visual feedback information to the user, and update according to the user's subsequent feedback dialogue status.

[0025] As described in S1-S8 above, the present invention aims to integrate multimodal data of user voice, facial expression video and text interaction through the integration of multimodal features such as acoustic, visual and text feature extraction and sentiment analysis to accurately identify the user's comprehensive emotional state and demand intentions, and then provide personalized interactive services including voice, text and visual feedback, thereby improving the accuracy of intention understanding and the fluency of interaction of the hotel intelligent dialogue system. In the hotel service scenario, user interaction information is complex, and it is difficult for single modal data to fully capture emotions and needs. Traditional single modal systems have problems of information loss and emotional misjudgment. This solution realizes the collaborative processing of multi-dimensional information by constructing a complete processing chain of "data acquisition-feature extraction-emotional analysis-demand identification-feedback generation". Specifically, multimodal raw data is first acquired through a microphone, camera, and text interface. The voice data is then subjected to noise reduction, framing, and Fourier transform processing to extract acoustic features and identify emotions. Key point localization, normalization, and CNN feature extraction are performed on facial expression videos to match emotion templates. Text data is segmented, word vectors are converted, and semantics are analyzed through LSTM to obtain emotional features. The three emotional feature dimensions are then aligned and weighted, and a rule engine and semantic parsing are combined to obtain user status and needs. The hotel service database is then searched based on the type of need and information is filtered according to user preferences and emotional state. The tone of the response is determined based on the comprehensive emotion. The service information is structured and integrated into multi-form feedback. Finally, the feedback is output synchronously and the conversation status is updated based on the user's subsequent interactions. This solution eliminates single-modal ambiguity through multimodal fusion and optimizes computing power requirements using lightweight algorithms. This improves the accuracy of emotion recognition and demand understanding, reduces interaction rounds, and shortens user waiting time.

[0026] In one embodiment, the step of obtaining user acoustic features based on the voice data and obtaining voice emotion features based on the user acoustic features includes: S201, preprocessing the voice data to obtain a preprocessed voice signal; S202, performing frame processing on the pre-processed voice signal to obtain multiple voice frames; S203, performing short-time Fourier transform on each of the speech frames to obtain a spectrogram corresponding to each of the speech frames; S204. Acquire user acoustic feature data according to each of the spectrograms, wherein the user acoustic feature data includes Mel-frequency cepstral coefficients, acoustic fundamental frequency feature data, and acoustic short-time energy feature data; S205, inputting the Mel-frequency cepstral coefficients, acoustic fundamental frequency feature data, and acoustic short-time energy feature data into a preset emotion recognition rule to obtain a comprehensive speech emotion feature; S206, obtaining preset emotion recognition rules according to the comprehensive speech emotion features; S207: Acquire user emotional state information according to the preset emotion recognition rule, where the user emotional state information includes positive state information, negative state information, and neutral state information.

[0027] In summary, this invention preprocesses the raw voice data captured by the microphone, using wavelet transform noise reduction or spectral subtraction to remove background noise (such as air conditioning and human voices) in the hotel environment. Endpoint detection is then used to eliminate silent segments, resulting in a pure preprocessed voice signal. This step aims to improve the signal-to-noise ratio of the voice signal, prevent noise from interfering with subsequent feature extraction, and lay the foundation for accurate acoustic feature analysis. The voice data is collected in real time by noise-reducing microphones deployed in the hotel and connected to the system processing module via a transmission link.

[0028] The preprocessed speech signal is divided into frames of appropriate length (e.g., using an overlapping framing strategy), breaking the continuous speech signal into multiple short-duration frames. Because speech signals are time-varying, framing converts them into short-duration stationary signals, facilitating subsequent frequency domain analysis. By properly setting the frame length and overlap ratio, the spectral characteristics within each frame are relatively stable, providing reliable input for the short-time Fourier transform (STFT) and improving the accuracy of feature extraction.

[0029] After applying a window function (such as a Hamming window) to each speech frame, a fast Fourier transform is performed to convert the time-domain speech signal into a frequency-domain spectrogram. The spectrogram visually displays the energy distribution of speech at different frequencies, providing a key basis for acoustic feature extraction. This step, through frequency-domain transformation, maps the timbre, pitch, and intensity of the speech into visual spectral features, facilitating the subsequent extraction of acoustic parameters related to emotion.

[0030] Multidimensional acoustic features are extracted from the spectrogram: Mel-frequency cepstral coefficients are obtained through a Mel filter bank and discrete cosine transform to characterize the spectral envelope of speech; the fundamental frequency is detected using autocorrelation or the YIN algorithm to reflect pitch variations; and short-term energy is calculated to characterize the intensity fluctuations of speech. These features capture emotional cues in speech from the dimensions of timbre, intonation, and dynamics, providing comprehensive physical signal characteristics for emotion recognition.

[0031] The extracted Mel-frequency cepstral coefficients, acoustic fundamental frequency feature data, and acoustic short-term energy feature data are fed into an emotion recognition rule base trained on hotel scenarios. This rule base uses a support vector machine (SVM) or hidden Markov model (HMM) to build a classifier. By learning the mapping between acoustic features and emotional states in historical speech data, it converts the current feature vector into a comprehensive speech emotion feature.

[0032] Based on the comprehensive speech emotion features, the system retrieves predefined emotion recognition rules (such as the feature threshold combinations corresponding to different emotional states) to ultimately determine the user's emotional state as "positive," "negative," or "neutral." Through scenario-based mapping within the rule library, the model's output feature space results are converted into understandable emotional semantic labels, providing a unified speech-dimensional emotional input for multimodal emotion fusion and ensuring that subsequent interactive feedback matches the user's emotional state.

[0033] In one embodiment, the step of obtaining user visual features based on the facial expression video data, and obtaining visual emotion features based on the user visual features, includes: S301, obtaining user facial key point positioning information based on the facial expression video data, wherein the user facial key point positioning information includes an eye area image, a mouth area image, and a facial muscle area image; S302, normalizing the user's facial key point positioning information, adjusting the image size and brightness contrast, and obtaining processed user's facial key point positioning information; S303, extracting features from the user's facial key point positioning information to obtain facial expression feature information and visual feature information; S304: Match the facial expression feature information and the visual feature information with a preset facial expression emotion template to obtain visual emotion features.

[0034] In summary, the present invention uses facial expression video frames captured by a camera to perform face detection, locate key points in the eyes, mouth, and key facial muscle areas, and then crop images of the eye, mouth, and facial muscle regions. This step focuses on the areas with the richest facial expressions, reducing background interference and providing precise regions of interest for subsequent feature extraction. The facial expression video data is collected in real time by high-definition cameras deployed in the hotel scene, ensuring the integrity of the expression details.

[0035] After keypoint localization, facial region images are uniformly resized to a fixed size, and brightness and contrast are optimized using a histogram equalization algorithm. This normalization process eliminates image size variations caused by lighting changes in hotel scenes (such as switching between natural and artificial light) and the user's distance from the camera. This ensures consistent input format for similar expression images under different conditions, providing stable visual input for subsequent feature extraction algorithms and preventing feature shifts caused by image quality fluctuations.

[0036] A lightweight convolutional neural network is used to extract features from the normalized regional images. The network automatically learns the key features of facial muscle deformation and expression texture through multiple layers of convolution and pooling operations, and generates a feature vector containing visual emotion information. The network parameters are optimized for the hotel scenario to make it more suitable for feature extraction of typical interaction expressions such as "concentrated expression during consultation" and "dissatisfied expression during complaint", while reducing the computational amount and improving the representation ability of expression features.

[0037] The extracted visual feature vector is matched with the facial expression emotion template library constructed based on FACS, and the corresponding emotion state (positive, negative, neutral) is determined according to a preset threshold. The template library is optimized for the expression patterns in common scenarios of user consultation and complaint in the hotel scenario. Through the pattern matching of the feature vector and the template, the abstract visual features are converted into understandable emotion semantic labels, providing accurate visual dimension emotion input for multi-modal emotion fusion.

[0038] In one embodiment, the step of obtaining user text features according to the text interaction data and obtaining text emotion features according to the user text features includes: S401. Perform word segmentation on the text interaction data to obtain a segmented text sequence; S402. Convert the text sequence into a word vector matrix to obtain user text features; S403. Perform semantic analysis on the user text features to obtain context semantic features; S404. Obtain text emotion features according to the context semantic features.

[0039] In summary, the present invention performs word segmentation on the text interaction data (such as chat messages, speech-to-text results) input by the user, removes punctuation marks and stop words such as "de" and "le", and obtains a structured segmented text sequence. This step converts unstructured text into a computable word sequence, facilitating subsequent semantic analysis. For the hotel scenario, a customized word segmenter can accurately segment professional terms such as "executive lounge" and "extra bed service", avoiding mis-segmentation problems of ordinary word segmentation tools and ensuring the integrity of domain vocabulary. The text data comes from the manual input of the user in the dialogue system or the text result of voice interaction.

[0040] The Word2Vec model is used to convert segmented text sequences into dense word embedding matrices, representing word meanings in a high-dimensional semantic space. To meet the needs of the hotel industry, a pre-trained model was used using a hotel service corpus containing terms like "room type" and "deposit." This approach places semantically similar words like "satisfaction" and "pleasure" closer together in the embedding space, addressing the "lexical independence" issue inherent in traditional one-hot encoding. This word embedding conversion captures implicit semantic associations between words. For example, the vector representations for "complaint" and "dissatisfaction" are highly similar, providing structured feature input for subsequent semantic analysis.

[0041] A long short-term memory (LSTM) network is used to perform sequence modeling on the word embedding matrix. A gating mechanism is used to capture long-range dependencies in the text (such as transitions and progressive logic) and extract feature vectors that reflect contextual semantics. For example, when analyzing the sentence "The room is clean but the soundproofing is poor," the LSTM can identify the transitional relationship between the word "but" and accurately extract negative sentiment cues. The network structure is optimized for hotel scenarios, using a single-layer LSTM architecture. Parameters are fine-tuned using hotel conversation data to improve semantic understanding of complex sentences such as "The front desk service is enthusiastic, but the facilities are outdated," avoiding misjudgments caused by analyzing word semantics in isolation.

[0042] The semantic feature vectors output by the LSTM are fed into a sentiment classifier based on a fully connected neural network. Combined with a hotel-specific sentiment lexicon (e.g., a vocabulary that labels "comfort" as positive and "noise" as negative), the probability of the text's sentiment tendency (positive, negative, or neutral) is calculated. The classifier parameters are trained using 200,000 pieces of annotated hotel scene data and optimized for typical comments such as "soft bedding" and "low service efficiency." Compared to general sentiment models, the accuracy of sentiment recognition in hotel settings is significantly improved. The resulting text sentiment features provide a numeric basis for sentiment in the text dimension for multimodal fusion, assisting the system in understanding the emotional tendencies of user needs.

[0043] In one embodiment, the step of fusing the voice emotion features, visual emotion features, and text emotion features to obtain multimodal features, and obtaining the user's comprehensive emotion state information and demand intention information based on the multimodal features includes: S501, dimensional alignment of the speech emotion features, visual emotion features, and text emotion features, S502, fusing the aligned speech emotion features, visual emotion features, and text emotion features to obtain multimodal features; S503: Obtain comprehensive status information of the user based on the multimodal features. S504, obtaining user emotion text features and intention text features according to the multimodal features; S505, obtaining comprehensive emotional state information according to the user's emotional text features; S506. Obtain user demand intention information according to the intention text feature, where the demand intention information includes hotel reservation, service consultation, and complaint suggestion.

[0044] In summary, the present invention unifies the emotional features of speech, vision, and text modalities into feature vectors of the same dimensionality through fully connected layer mapping or principal component analysis (PCA). Because the dimensionality and representational space of the original features of different modalities differ (for example, visual features are generally higher in dimensionality than speech), dimensional alignment ensures comparability of features across modalities within the same semantic space, providing a foundation for subsequent fusion processing. This step achieves dimensionality unification through linear transformation or dimensionality reduction algorithms while preserving the primary feature variance, thus avoiding feature fusion bias caused by dimensional inconsistency.

[0045] A dynamic weighted summation strategy is used to fuse the dimensionally aligned trimodal features, with weights dynamically adjusted based on each modality's historical accuracy in hotel scenarios. This dynamic weighting mechanism adaptively matches the importance of each modality in different interaction contexts (for example, the voice modality's weight is automatically increased during voice interaction), addressing the problem of traditional static weighting being unable to adapt to changing scenarios. The fused multimodal features integrate complementary information from each modality, enabling a more comprehensive representation of user emotions and needs, providing rich feature input for subsequent semantic analysis.

[0046] The fused multimodal features are fed into the comprehensive state analysis module based on a rule engine. Combined with customized emotion fusion rules for hotel scenarios, conditional judgments are performed to generate a probability distribution of the user's comprehensive emotional state. The rule engine includes built-in multimodal emotion association rules tailored to hotel interaction scenarios (such as joint judgment logic for voice and visual features), which effectively eliminate ambiguity in single modalities and improve the accuracy of emotional state judgments. This step, through the joint analysis of multi-source features, achieves a comprehensive understanding of the user's emotional state, providing an emotionally informed decision-making basis for subsequent service responses.

[0047] A bidirectional long short-term memory (Bi-LSTM) network is used to perform sequence modeling of multimodal features. Combined with a hotel domain terminology library, text features related to sentiment and demand intent are isolated. Through semantic feature decoupling, sentiment cues (such as "satisfied" and "dissatisfied") and intent keywords (such as "booking" and "complaint") are extracted separately from the multimodal fusion features, facilitating the subsequent generation of targeted sentiment and demand intent information. This step leverages the integration of deep learning models and domain knowledge to enhance the ability to parse hotel terminology and complex semantics.

[0048] In one embodiment, the step of obtaining hotel service information related to user needs based on the demand intention information includes: S601: Acquire a user demand type according to the demand intention information, wherein the demand type includes room reservation, catering service, and facility inquiry; S602: Acquire hotel service data according to the demand type, and retrieve corresponding hotel service information according to the hotel service data, wherein the hotel service database includes room type, price, restaurant menu, and facility opening hours data.

[0049] In summary, the present invention uses keyword matching and semantic parsing algorithms to map demand intent information derived from multimodal fusion into a pre-defined hotel demand typology (including room reservations, food and beverage services, and facility inquiries). Specifically, a hotel keyword library (e.g., "room type" and "breakfast") is combined with a forward maximum matching algorithm to identify demand keywords. For requests without explicit keywords (e.g., "quiet room"), an LSTM model is used to analyze contextual semantics for classification. This step converts user natural language requests into structured categories, addressing the crude nature of traditional classification and providing precise demand guidance for subsequent service information retrieval.

[0050] Based on the type of demand, corresponding information is retrieved from the hotel service database. The database uses a relational architecture (such as MySQL) and includes table structures for room types and dining menus. It synchronizes with the hotel's PMS system in real time via an API to ensure information timeliness. During retrieval, SQL query statements are generated based on the user's historical preferences and current emotional state (for example, filtering available services by demand type). Retrieval efficiency is improved through index optimization and caching mechanisms. This step addresses the information lag and matching ambiguity issues in traditional retrieval, enabling rapid matching between "demand intent and service information." Retrieval results are optimized and sorted based on user profile and emotional state to ensure accurate and timely interactive responses.

[0051] In one embodiment, the step of obtaining user feedback information based on the comprehensive emotional state information and hotel service information includes: S701, obtaining preset reply tone information according to the comprehensive emotional state; S702: Generate structured reply content based on the hotel service information, wherein the structured reply content includes a text description, a picture link, and a speech synthesis text; S703: Integrate the preset reply tone information and the structured reply content to generate voice reply, text reply and visual feedback information; S704: Obtain user feedback information based on the voice reply, text reply, and visual feedback information.

[0052] In summary, the present invention maps the user's comprehensive emotional state (positive, negative, neutral) obtained through multimodal fusion to a preset tone template library to obtain the corresponding reply tone. Positive emotions are matched with enthusiastic and lively tone parameters (such as speaking speed and rising intonation), negative emotions are matched with soothing and sincere tone styles (such as slowing speaking speed), and neutral emotions use an objective and stable tone mode. This step enhances emotional resonance in the interaction through dynamic matching of emotion and tone. For example, when the user is in a negative state, the system automatically calls the soothing tone template, alleviates the user's dissatisfaction through tone adjustment, and improves feedback acceptance.

[0053] Based on hotel service information search results, structured content is generated, including text descriptions, image links, and speech synthesis text. Specifically, key data (such as room type and price) is extracted from the service information and combined with pre-set templates to generate natural language descriptions. Links are generated to corresponding image resources based on the service type. The text descriptions are then converted into speech synthesis text with punctuation and accent cues. Structured content reduces user comprehension costs by complementing multiple forms of information. For example, restaurant service search results can generate dish descriptions, image links, and speech synthesis text, achieving a multi-dimensional presentation of information.

[0054] The preset response tone and structured content are integrated into a voice, text, and visual feedback output. Voice responses use text-to-speech (TTS) processing based on tone parameters (e.g., lowering the tone in negative situations). Text responses combine text and image links into rich text. Visual feedback generates corresponding icons (e.g., smiley faces, apology symbols) based on emotional state and combines them with images. Multimodal integration adapts to different user information preferences through a synchronized output mechanism (e.g., simultaneous presentation within 0.5 seconds). For example, voice-first users can access information through TTS, while visual users can quickly understand service content through images.

[0055] The system captures voice responses through a microphone, monitors text interactions (such as image clicks), and records visual element manipulation to obtain real-time user feedback on the content. After noise reduction and semantic parsing, the collected feedback information is generated into feedback labels such as "interested" and "uncomprehended." These labels are stored in the conversation state database and provide a basis for subsequent conversation state updates. For example, when a user clicks an image link, the system identifies it as "interested" and proceeds with the booking process. If there is no response, a secondary feedback strategy is triggered to ensure the consistency and adaptability of the interaction logic. Example 2

[0056] This application also provides a hotel intelligent voice dialogue system based on multimodal interaction, such as Figure 2 Shown, including: Data collection module 1 is used to obtain multimodal data of users in hotel voice conversations, wherein the multimodal data includes voice data, facial expression video data and text interaction data; A speech emotion feature extraction module 2 is used to obtain user acoustic features based on the speech data, and to obtain speech emotion features based on the user acoustic features; A visual emotion feature extraction module 3 is used to obtain user visual features based on the facial expression video data, and to obtain visual emotion features based on the user visual features; A text sentiment feature extraction module 4 is configured to obtain user text features based on the text interaction data, and to obtain text sentiment features based on the user text features; Analysis module 5, used to fuse the voice emotion features, visual emotion features, and text emotion features to obtain multimodal features, and obtain the user's comprehensive emotional state information and demand intention information based on the multimodal features; An information retrieval module 6 is used to obtain hotel service information related to user needs based on the demand intention information; A multimodal feedback generation module 7 is configured to obtain user feedback information based on the comprehensive emotional state information and hotel service information, wherein the feedback information includes voice response, text response and visual feedback information; The interactive output and dialogue state management module 8 is used to synchronously output the voice reply, text reply and visual feedback information to the user, and update the dialogue state according to the user's subsequent feedback.

[0057] In one embodiment, the speech emotion feature extraction module 2 includes: A speech signal preprocessing unit, configured to preprocess the speech data to obtain a preprocessed speech signal; A voice signal framing processing unit, configured to perform framing processing on the pre-processed voice signal to obtain a plurality of voice frames; a transform processing unit, configured to perform a short-time Fourier transform on each of the speech frames to obtain a spectrogram corresponding to each of the speech frames; an acoustic feature extraction unit, configured to obtain user acoustic feature data according to each of the spectrograms, wherein the user acoustic feature data includes Mel-frequency cepstral coefficients, acoustic fundamental frequency feature data, and acoustic short-time energy feature data; An emotion recognition rule application unit is used to input the Mel-frequency cepstral coefficients, acoustic fundamental frequency feature data and acoustic short-time energy feature data into a preset emotion recognition rule to obtain a comprehensive speech emotion feature; An emotion preset rule acquisition unit, configured to acquire an emotion preset recognition rule based on the comprehensive speech emotion feature; The state determination unit is used to obtain user emotional state information according to the preset emotion recognition rule, and the user emotional state information includes positive state information, negative state information and neutral state information. Example 3

[0058] like Figure 3 As shown, embodiment 3 of the present invention provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the hotel intelligent voice dialogue method based on multimodal interaction provided in embodiment 1 is implemented.

[0059] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A hotel intelligent voice dialogue method based on multimodal interaction, characterized in that: include: Acquire multimodal data of the user's voice conversation in the hotel, wherein the multimodal data includes voice data, facial expression video data, and text interaction data; Acquire user acoustic features based on the voice data, and acquire voice emotion features based on the user acoustic features; acquire user visual features based on the facial expression video data, and acquire visual emotion features based on the user visual features; acquire user text features based on the text interaction data, and acquire text emotion features based on the user text features; The speech emotion feature, visual emotion feature, and text emotion feature are integrated to obtain a multimodal feature, and the user's comprehensive emotion state information and demand intention information are obtained based on the multimodal feature; Acquire hotel service information related to user needs based on the demand intention information; Obtaining user feedback information based on the comprehensive emotional state information and hotel service information, wherein the feedback information includes voice response, text response, and visual feedback information; The voice reply, text reply and visual feedback information are synchronously output to the user and updated according to the user's subsequent feedback dialogue status.

2. A hotel intelligent voice dialogue method based on multimodal interaction according to claim 1, characterized in that, The step of obtaining user acoustic features based on the voice data, and obtaining voice emotion features based on the user acoustic features, includes: Preprocessing the voice data to obtain a preprocessed voice signal; Performing frame processing on the preprocessed speech signal to obtain a plurality of speech frames; Performing a short-time Fourier transform on each of the speech frames to obtain a spectrogram corresponding to each of the speech frames; Acquiring user acoustic feature data according to each of the spectrograms, wherein the user acoustic feature data includes Mel-frequency cepstral coefficients, acoustic fundamental frequency feature data, and acoustic short-time energy feature data; Inputting the Mel-frequency cepstral coefficients, acoustic fundamental frequency feature data and acoustic short-time energy feature data into a preset emotion recognition rule to obtain a comprehensive speech emotion feature; Acquiring emotion preset recognition rules according to the comprehensive speech emotion features; The user's emotional state information is acquired according to the preset emotion recognition rule, where the user's emotional state information includes positive state information, negative state information, and neutral state information.

3. A hotel intelligent voice dialogue method based on multimodal interaction according to claim 1, characterized in that, The step of obtaining user visual features based on the facial expression video data, and obtaining visual emotion features based on the user visual features, comprises: Acquire user facial key point positioning information based on the facial expression video data, wherein the user facial key point positioning information includes an eye area image, a mouth area image, and a facial muscle area image; Normalizing the user's facial key point positioning information, adjusting the image size and brightness contrast, and obtaining processed user's facial key point positioning information; Extract features from the user's facial key point positioning information to obtain facial expression feature information and visual feature information; The facial expression feature information and visual feature information are matched with a preset facial expression emotion template to obtain visual emotion features.

4. A hotel intelligent voice dialogue method based on multimodal interaction according to claim 1, characterized in that, The step of obtaining user text features based on the text interaction data, and obtaining text sentiment features based on the user text features, includes: Performing word segmentation processing on the text interaction data to obtain a word segmentation text sequence; Convert the text sequence into a word vector matrix to obtain user text features; Performing semantic analysis on the user text features to obtain contextual semantic features; The text sentiment feature is obtained according to the context semantic feature.

5. A hotel intelligent voice dialogue method based on multimodal interaction according to claim 1, characterized in that, The step of fusing the voice emotion feature, visual emotion feature, and text emotion feature to obtain a multimodal feature, and obtaining the user's comprehensive emotion state information and demand intention information based on the multimodal feature, includes: Align the dimensions of the speech emotion features, visual emotion features, and text emotion features, The aligned speech emotion features, visual emotion features, and text emotion features are fused to obtain multimodal features; Obtaining comprehensive status information of the user according to the multimodal features, Acquiring user emotion text features and intention text features based on the multimodal features; Acquiring comprehensive emotional state information based on the user's emotional text features; The user demand intention information is obtained according to the intention text features, and the demand intention information includes hotel reservation, service consultation, and complaint suggestions.

6. A hotel intelligent voice dialogue method based on multimodal interaction according to claim 1, characterized in that, The step of obtaining hotel service information related to user needs according to the demand intention information includes: Acquiring a user demand type according to the demand intention information, wherein the demand type includes room reservation, catering service, and facility inquiry; Acquire query hotel service data according to the demand type, and retrieve corresponding hotel service information according to the query hotel service data, wherein the hotel service database includes room type, price, restaurant menu, and facility opening hours data.

7. A hotel intelligent voice dialogue method based on multimodal interaction according to claim 1, characterized in that, The step of obtaining user feedback information based on the comprehensive emotional state information and hotel service information includes: Obtaining preset response tone information according to the comprehensive emotional state; Generating structured reply content according to the hotel service information, wherein the structured reply content includes text description, image link and speech synthesis text; Integrating the preset reply tone information and the structured reply content to generate voice reply, text reply and visual feedback information; User feedback information is obtained based on the voice response, text response and visual feedback information.

8. The hotel intelligent voice dialogue system based on multimodal interaction is characterized by: include: A data acquisition module is used to obtain multimodal data of users' voice conversations in the hotel, wherein the multimodal data includes voice data, facial expression video data and text interaction data; A speech emotion feature extraction module, configured to obtain user acoustic features based on the speech data, and to obtain speech emotion features based on the user acoustic features; A visual emotion feature extraction module, configured to obtain user visual features based on the facial expression video data, and to obtain visual emotion features based on the user visual features; A text sentiment feature extraction module, configured to obtain user text features based on the text interaction data, and to obtain text sentiment features based on the user text features; An analysis module is used to fuse the voice emotion features, visual emotion features, and text emotion features to obtain multimodal features, and obtain the user's comprehensive emotional state information and demand intention information based on the multimodal features; An information retrieval module is used to obtain hotel service information related to user needs based on the demand intention information; a multimodal feedback generation module, configured to obtain user feedback information based on the comprehensive emotional state information and hotel service information, wherein the feedback information includes voice response, text response, and visual feedback information; The interactive output and dialogue state management module is used to synchronously output the voice reply, text reply and visual feedback information to the user, and update the dialogue state according to the user's subsequent feedback.

9. The hotel intelligent voice dialogue system based on multimodal interaction according to claim 8, characterized in that: The speech emotion feature extraction module includes: A speech signal preprocessing unit, configured to preprocess the speech data to obtain a preprocessed speech signal; A voice signal framing processing unit, configured to perform framing processing on the pre-processed voice signal to obtain a plurality of voice frames; a transform processing unit, configured to perform a short-time Fourier transform on each of the speech frames to obtain a spectrogram corresponding to each of the speech frames; an acoustic feature extraction unit, configured to obtain user acoustic feature data according to each of the spectrograms, wherein the user acoustic feature data includes Mel-frequency cepstral coefficients, acoustic fundamental frequency feature data, and acoustic short-time energy feature data; An emotion recognition rule application unit, configured to input the Mel-frequency cepstral coefficients, acoustic fundamental frequency feature data, and acoustic short-time energy feature data into a preset emotion recognition rule to obtain a comprehensive speech emotion feature; An emotion preset rule acquisition unit, configured to acquire an emotion preset recognition rule based on the comprehensive speech emotion feature; The state determination unit is used to obtain user emotional state information according to the preset emotion recognition rule, and the user emotional state information includes positive state information, negative state information and neutral state information.

10. A computer-readable storage medium, characterized in that A program is stored thereon, and when the program is executed by a processor, the hotel intelligent voice dialogue method based on multimodal interaction described in any one of claims 1 to 7 is implemented.

Images