A method and system for generating personalized voice guides based on tourists' emotional states

Abstract

This invention discloses a method and system for generating personalized voice guides based on tourists' emotional states, relating to the fields of state recognition and speech generation technology. It involves collecting multi-dimensional data reflecting tourists' real-time emotional states and interests, preprocessing the collected multi-dimensional data to extract visual and physiological modal features, constructing and training an emotion recognition model based on these features to output tourists' emotional state labels, calculating tourists' interest focus labels based on their visual modal features and points of interest, dynamically selecting and organizing information from a knowledge graph based on tourists' real-time emotional states and interest focuses to generate personalized narrative text, and converting the personalized narrative text into speech for broadcast. This invention, employing the aforementioned method and system for generating personalized voice guides based on tourists' emotional states, can continuously optimize the output according to tourists' emotional states and interest focuses, adapting to the personalized needs of different tourists.

Description

Technical Field

[0001] This invention relates to the fields of state recognition and speech generation technology, and in particular to a method and system for generating personalized voice guides based on tourists' emotional states. Background Technology

[0002] The current mainstream smart tour guide technologies mainly include: pre-recorded audio tours, where tourists trigger the playback of fixed audio files through location; text-to-speech (TTS) tours, which play fixed text information through mechanical speech synthesis technology; and simple interactive question-and-answer robots that reply with preset answers based on keyword matching.

[0003] Pre-recorded or fixed TTS content cannot be dynamically adjusted according to tourists' real-time interests and emotional states, resulting in a monotonous and generic experience. Existing guided tours operate on a one-way information delivery model of "play-listen," failing to perceive tourists' emotional feedback (such as confusion, excitement, or boredom) and unable to establish a warm, engaging narrative dialogue. Traditional TTS voice is mechanical and bland, lacking rhythm, emotion, and a sense of audience, making it difficult to fulfill the functions of high-quality cultural narrative and emotional communication, falling far short of the artistic effects pursued by broadcasting and hosting. The root cause lies in the system's lack of real-time, accurate perception of tourists' states and its lack of ability to generate and interpret high-quality, dynamic content based on perceived information. Technology and art (broadcasting and narration) are disconnected; multimodal emotion computing (judging emotional states through the fusion of multi-dimensional data such as images, speech, and physiological signals); deep learning-driven natural language generation (NLG); and emotional speech synthesis (Emotional TTS, which can control the timbre, tone, and rhythm of speech to express different emotions). Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for generating personalized voice guides based on tourists' emotional states. This method accurately reflects tourists' real-time emotional states and interests through multi-dimensional data collection, constructs an emotion recognition model to calculate and output tourists' emotional states, calculates tourists' focus of interest based on their visual modal characteristics and points of interest (POIs), and generates personalized narrative text that is context-appropriate, of suitable length, and emotionally matched based on "emotion-interest," which is then converted into voice broadcasts. This provides more personalized voice services for tourist areas, attractions, and other service scenarios, attracting more visitors, increasing tourist numbers, driving surrounding economic development, and attracting corporate investment.

[0005] To achieve the above objectives, the present invention provides a method and system for generating personalized voice guides based on tourists' emotional states, comprising the following steps: S1. Collect multi-dimensional data reflecting tourists' real-time emotional state and interest preferences; S2. After preprocessing the multi-dimensional data collected in S1, visual modal features and physiological modal features are extracted. S3. Based on the visual modal features and physiological modal features of S2, construct and train an emotion recognition model to output the emotional state labels of tourists; S4. Calculate the tourist's interest focus tags based on the tourist's visual modal characteristics and points of interest; S5. Based on tourists' real-time emotional state and focus of interest, dynamically select and organize information from the knowledge graph to generate personalized narrative text that is context-appropriate, of suitable length, and emotionally resonant. S6. Convert the personalized narrative text generated in S5 into speech for broadcast.

[0006] Preferably, the devices for collecting multi-dimensional data in S1 include high-definition cameras deployed in key points of interest areas of the attraction to collect tourists' facial data, and lightweight wearable devices distributed to tourists upon entry to collect tourists' physiological data.

[0007] Preferably, the specific process of S2 is as follows: S21. Remove outliers to complete data cleaning and use interpolation to fill in temporary data gaps; S22. Based on timestamps, facial data and physiological data are aligned to complete data synchronization and construct a unified multimodal data time series; S23. Extract visual modal features. Extract the coordinates of key facial feature points using a multi-task convolutional neural network (MTCNN) and calculate the geometric features between the coordinates of key facial feature points. Calculate the gaze focus area based on the spatial intersection of the gaze ray and the coordinates of scene interest points. S24. Extract physiological modal features by performing frequency domain analysis and smoothing on physiological data to extract peak and mean features; S25. The extracted visual modal features and physiological modal features are normalized using the Z-score normalization method. The specific calculation formula is as follows: ; in, For the first i The first time point j 1 eigenvalue, For the first j The mean of each feature, For the first j The standard deviation of each feature.

[0008] Preferably, the S3 emotion recognition model adopts a three-level architecture: a single-modal feature encoding layer, a cross-modal feature fusion layer, and an emotion classification layer, as follows: S31. Design a single-modal feature coding layer that includes visual modality coding and physiological modality coding; Visual modality encoding uses a convolutional neural network (CNN) architecture. The input is a temporal sequence of key facial feature points. Local spatiotemporal features are extracted through three convolutional layers, and temporal dependencies are captured through an LSTM layer to output a visual modality feature vector. Physiological modality coding: A multilayer perceptron (MLP) architecture is adopted. The input is the standardized physiological modality features, which are mapped through three fully connected layers to output physiological modality feature vectors. S32. Design a cross-modal feature fusion layer, using an attention mechanism to achieve adaptive fusion of multimodal features, and calculate the association weights between visual modal feature vectors and physiological modal feature vectors, as shown in the following formula: ; ; in, For visual modality feature vectors, This is a feature vector representing physiological modalities. Attention weights for visual features This is the weight matrix. For bias terms, The square root of the visual feature dimension is used to dynamically adjust the contribution of feature vectors of different modalities. S33. Design the sentiment classification layer, using an MLP architecture, with the input being a fused feature vector. Through two fully connected layers and a Dropout layer, the probability distribution of various emotions is finally output through the Softmax activation function, as shown in the following formula: ; in, For the emotion category, among which =4 and When =1 corresponds to curiosity. =2 corresponds to focus. =3 corresponds to pleasure. =4 corresponds to fatigue. For the first Class weight matrix, For the first Class bias terms, The weight matrix is ​​responsible for integrating the fused multimodal features. Mapped to the Emotional type for The accompanying bias term is used to adjust the decision boundary of sentiment classification, taking the category with the highest probability as the final sentiment state label; S34. Output the tourist's emotional state label and corresponding confidence level.

[0009] Preferably, the specific training process of the emotion recognition model in S3 is as follows: The training dataset combines a public sentiment dataset with a custom tourist attraction dataset. The custom tourist attraction dataset contains multimodal data from 1,000 tourists of different ages and genders, labeled with four sentiment tags. The cross-entropy loss function is used, and the formula is as follows: ; in, The number of training samples, For the first The true label of each sample For the first The predicted value for the nth sample is... The probability of similar emotions; The Adam optimizer was used, with an initial learning rate of 0.001, which was reduced to 0.9 every 10 epochs. The training iterations were performed for 100 epochs until the loss function converged.

[0010] Preferably, the specific process for calculating the focus of interest in S4 is as follows: S41. Calculate the spatial equation of the line-of-sight ray based on pupil position data from visual modal features acquired by the camera: ; in, , , Associate each point of interest with a unique range of spatial coordinates. 3D coordinates of the visitor's pupils The direction vector of the line of sight is calculated by connecting the pupil and the feature point at the corner of the eye. S42. Perform interest point association judgment. Determine whether the line of sight intersects with the spatial coordinate range of a certain interest point. If they intersect, record the interest point as a candidate interest focus. If they do not intersect, do not record it. To determine whether a line of sight intersects the spatial coordinate range of a point of interest, a spatial geometric intersection algorithm is used, as shown in the following formula: Let the spatial extent of the point of interest be a cuboid: , , The parametric equation for the line of sight ray is: ; Solve the following system of inequalities: ; in, Here, represents the distance extended along the line of sight from the pupil; if it exists... If the solution is not found, then determine whether the line of sight intersects with the point of interest; if not, then... If the solution is correct, then it is determined that the line of sight does not intersect with the point of interest; S43. Calculate the time tourists spend looking at candidate focal points of interest. Interest scores are calculated by combining the confidence level of tourists’ focus on emotional tags; Calculate tourists' interest scores for candidate focus areas of interest. as follows: ; in, Based on the base stay time, To focus on the confidence level of sentiment labels, , all These are the weighting coefficients. The candidate focus of interest is then determined as the current focus of interest. S44. Recalculate the interest score every 500ms. If the score of the new candidate interest focus exceeds the current interest focus by more than 0.2, update the interest focus label; otherwise, keep the current interest focus label. S45. Output the tourist's current interest focus tags and interest score, and record the switching frequency of interest focus tags.

[0011] Preferably, S5 adopts a three-layer architecture of knowledge graph support, narrative logic planning, and natural language text generation. Each layer works together to achieve personalized language text generation, as detailed below: S51. Define a knowledge graph for the tourism sector, with attractions and points of interest as core nodes, construct a multi-relationship, multi-level structured knowledge network, including entity nodes, relationship types, attribute information, and provide knowledge graph storage and retrieval functions; S52. Emotion-interest driven narrative logic planning: constructing a narrative logic rule base and using dynamic programming algorithm to select the optimal narrative structure; S53. It adopts a fine-tuning architecture based on a pre-trained language model. The input is interest focus labels, sentiment state labels, and narrative structure prompts, and the output is natural language text. S54. Perform length verification and syntax correction optimization on the output natural language text, and finally output personalized narrative text that meets the requirements.

[0012] Preferably, the specific process of converting the generated personalized narrative text into speech with corresponding emotional color and artistic expression in S6 for broadcasting is as follows: S61. Design an emotion mapping and speech generation mechanism to map abstract emotion state labels into quantifiable acoustic parameters, driving the speech synthesis model to generate speech that conforms to emotional features. S62. Select the core acoustic parameters that affect the expression of emotion in speech, and use a multiple linear regression model to establish the mapping from emotional state labels to acoustic parameters; S63. The speech synthesis model adopts an end-to-end Transformer-TTS architecture, introduces emotional acoustic parameter constraints in the acoustic decoder, and uses the mapped acoustic parameters as the conditional input of the acoustic decoder so that the generated Mel spectrum meets the requirements of emotional features. S64. Perform noise reduction and volume normalization on the generated speech waveform; S65. Outputs voice through multiple terminals and updates acoustic parameters in real time after the tourist's emotional state changes, reflecting the emotional switch in the next voice segment to ensure real-time matching between voice and tourist's state.

[0013] Preferably, the mapping process of S62 is as follows: S621. Suppose that the emotional state labels are converted into vectors through one-hot encoding. , , , , Corresponding to curiosity, focus, pleasure, and fatigue respectively, the acoustic parameter vectors are: The mapping relationship is then: ; in, The weight matrix is ​​5×4, representing the influence weight of each emotional state on each acoustic parameter. It is a 5-dimensional bias vector. This is the random error term; S622. Using an emotional speech corpus, solve the problem using the least squares method. and The formula for minimizing the mean square error (MSE) between the predicted and actual acoustic parameters is as follows: ; in, For the first The true acoustic parameters of each sample For the first The sentiment vector of each sample.

[0014] To achieve the above objectives, the present invention also provides a personalized voice guide generation system based on tourists' emotional states. The system includes a multi-dimensional data acquisition module, a preprocessing module, an emotion recognition module, an interest focus calculation module, a text generation module, and a voice conversion module. The multi-dimensional data acquisition module is responsible for facial data acquisition and physiological data acquisition; The preprocessing module is responsible for cleaning and synchronizing facial and physiological data, extracting features, and standardizing the data. The emotion recognition module is responsible for single-modal feature encoding, cross-modal feature fusion, and emotion classification, outputting the emotional state labels and confidence scores of tourists. The interest focus calculation module is responsible for determining the candidate interest focus of tourists by associating points of interest and calculating the interest score of the candidate interest focus to update the candidate interest focus; The text generation module is responsible for providing knowledge graph storage and retrieval, planning narrative logic to output natural language text, and then optimizing the natural language text to generate personalized narrative text. The speech conversion module is responsible for designing the emotion mapping and speech generation mechanism to generate speech that conforms to emotional characteristics.

[0015] Therefore, the personalized voice guide generation method and system based on tourist emotional state described above has the following advantages compared with the prior art: 1. This application achieves truly personalized storytelling by sensing tourists' real-time emotions and interests, dynamically generating and broadcasting matching content, transforming guided tours from "standardized products" into "personalized services," significantly enhancing tourists' immersion and satisfaction; 2. This application creates a warm interactive experience. The system has "perception-response" capabilities, which can adjust the pace of the content according to the fatigue of tourists and provide in-depth interpretations according to their doubts. It simulates the empathy and interaction capabilities of an excellent tour guide or host, and enhances emotional resonance. 3. It enhances the artistry of cultural dissemination. Through emotional TTS technology, techniques such as "objectivity," "scene reproduction," and "inner language" from the art of broadcasting and hosting are integrated into the broadcast as technical parameters, making the audio guide more infectious, story-driven, and artistically expressive, and better realizing the transmission of cultural values. It also improves the intelligence and adaptability of the system, which can continuously optimize the output based on tourist feedback (emotional state) to adapt to the personalized needs of different tourists, and has high technical barriers.

[0016] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0017] Figure 1 This is an overall flowchart of the personalized voice guide generation method based on tourists' emotional states according to the present invention; Figure 2 This is a structural diagram of the personalized voice guide generation system based on the emotional state of tourists, as described in this invention. Detailed Implementation

[0018] In the description of this invention, it should be noted that the terms "upper," "lower," "inner," "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings, or the orientation or positional relationship in which the product of this invention is usually placed when in use. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limiting this invention.

[0019] Example like Figure 1 As shown, the personalized voice guide generation method and system based on tourist emotional state of the present invention includes the following steps: S1. Collect multi-dimensional data reflecting tourists' real-time emotional state and interest preferences; The devices used to collect multi-dimensional data include high-definition cameras deployed in key points of interest areas of the attraction to collect tourists' facial data, and lightweight wearable devices distributed to tourists upon entry to collect tourists' physiological data; providing high-quality and timely raw input for subsequent analysis. Data collection must meet three core requirements: non-invasiveness, continuity, and low latency, as shown in Table 1.

[0020] Table 1. Data acquisition devices and corresponding data types;

[0021] S2. After preprocessing the multi-dimensional data collected in S1, visual modal features and physiological modal features are extracted. S21. Remove outliers (such as invalid facial data caused by camera obstruction, physiological signal spikes caused by accidental device touch) to complete data cleaning and use interpolation to fill in temporary data gaps (gap duration ≤ 2s). S22. Based on timestamps, align facial data and physiological data to achieve data synchronization (time error ≤ 50ms) and construct a unified multimodal data time series; S23. Extract visual modal features. Extract the coordinates of 68 key facial feature points using a multi-task convolutional neural network (MTCNN). Calculate the geometric features such as distance and angle between the coordinates of the key facial feature points (e.g., the vertical distance between the eyebrow peak and the corner of the eye, and the upward angle of the corner of the mouth). Calculate the gaze focus area based on the spatial intersection of the gaze ray and the coordinates of the scene interest points. S24. Extract physiological modal features. This involves performing frequency domain analysis and smoothing on physiological data to extract peak and mean features. Specifically, frequency domain analysis (FFT transform) is performed on HRV data to extract low-frequency components (LF, 0.04-0.15Hz), high-frequency components (HF, 0.15-0.4Hz), and the LF / HF ratio. EDA data is smoothed (moving average window = 5s) to extract peak and mean features. S25. The extracted visual modal features and physiological modal features are normalized using the Z-score normalization method. The specific calculation formula is as follows: ; in, For the first i The first time point j 1 eigenvalue, For the first j The mean of each feature, For the first j The standard deviation of each feature.

[0022] S3. Based on the visual modal features and physiological modal features of S2, construct and train an emotion recognition model to output the emotional state labels of tourists; The emotion recognition model adopts a three-level architecture: a single-modal feature encoding layer, a cross-modal feature fusion layer, and an emotion classification layer, as detailed below: S31. Design a single-modal feature coding layer that includes visual modality coding and physiological modality coding; Visual modality encoding employs a convolutional neural network (CNN) architecture. The input is a temporal sequence of key facial feature points (dimension: T×68×2, where T is the length of the time window, taking 10 frames). Local spatiotemporal features are extracted through three convolutional layers (3×3 kernel size, stride 1, same padding). Then, temporal dependencies are captured through an LSTM layer (128 hidden units), and the output is a visual modality feature vector (dimension: 256). Physiological modality encoding: A multilayer perceptron (MLP) architecture is adopted. The input is the standardized physiological modality features (dimension: 64, including 16 types of features such as HRV LF, HF, LF / HF, EDA peak and mean, etc. × 4 time window statistics). The features are mapped through 3 fully connected layers (128, 64 and 32 neurons respectively) and the output is a physiological modality feature vector (dimension: 32). S32. Design a cross-modal feature fusion layer, using an attention mechanism to achieve adaptive fusion of multimodal features, and calculate the association weights between visual modal feature vectors and physiological modal feature vectors, as shown in the following formula: ; ; in, For visual modality feature vectors, This is a feature vector representing physiological modalities. Attention weights for visual features (values ​​range from 0 to 1). This is the weight matrix. For bias terms, This is the square root of the visual feature dimension. This mechanism allows for the dynamic adjustment of the contribution of feature vectors from different modalities (e.g., when facial expressions are clear). When the value approaches 1, physiological signals are stable. (approaching 0.5). S33. Design the sentiment classification layer, using an MLP architecture, with the input being a fused feature vector. The algorithm uses two fully connected layers (128 and 64 neurons respectively) and a Dropout layer (dropout rate = 0.3 to prevent overfitting). Finally, it outputs the probability distribution of various emotions through the Softmax activation function, as shown in the following formula: ; in, For the emotion category, among which =4 and When =1 corresponds to curiosity. =2 corresponds to focus. =3 corresponds to pleasure. =4 corresponds to fatigue. For the first Class weight matrix, For the first Class bias terms, The weight matrix is ​​responsible for integrating the fused multimodal features. Mapped to the Emotional type for The accompanying bias term is used to adjust the decision boundary of sentiment classification, taking the category with the highest probability as the final sentiment state label; S34. Output the tourist's emotional state label (value set: {curious, focused, happy, tired}) and the corresponding confidence level (a corresponding confidence level ≥ 0.7 is considered a valid label, and if it is lower than 0.7, take the label from the previous moment and mark it as pending confirmation).

[0023] The specific training process of the emotion recognition model is as follows: The training dataset combines publicly available sentiment datasets (such as AffectNet and DEAP) with a custom tourist dataset. The custom tourist dataset contains multimodal data from 1,000 tourists of different ages and genders (20 minutes per person, covering different tourist scene scenarios), labeled with 4 sentiment tags. The cross-entropy loss function is used, and the formula is as follows: ; in, The number of training samples, For the first The true labels of each sample (one-hot encoding) For the first The predicted value for the nth sample is... The probability of similar emotions; The Adam optimizer was used, with an initial learning rate of 0.001, which was reduced to 0.9 every 10 epochs. The training iterations were 100 epochs, until the loss function converged (the difference in loss between two consecutive epochs was ≤1e-5).

[0024] S4. Calculate the tourist's interest focus tags based on the tourist's visual modal characteristics and points of interest; Points of Interest (POIs) refer to the specific objects within a tourist attraction that are currently of interest to visitors. These POIs are predefined as entities within the attraction that can be independently explained (e.g., artifact A, architectural detail B, rest area C, etc.). Each POI is associated with a unique spatial coordinate range (e.g., the coordinate range of artifact A: ...). [10,20]、 [30,40]、 [0,5], unit: meters); The specific process for calculating the focus of interest in S4 is as follows: S41. Calculate the spatial equation of the line-of-sight ray based on pupil position data from visual modal features acquired by the camera: ; in, , , Associate each point of interest with a unique range of spatial coordinates. 3D coordinates of the visitor's pupils The direction vector of the line of sight is calculated by connecting the pupil and the feature point at the corner of the eye. S42. Perform interest point association judgment. Determine whether the line of sight intersects with the spatial coordinate range of a certain interest point. If they intersect, record the interest point as a candidate interest focus. If they do not intersect, do not record it. To determine whether a line of sight intersects the spatial coordinate range of a point of interest, a spatial geometric intersection algorithm is used, as shown in the following formula: Let the spatial extent of the point of interest be a cuboid: , , The parametric equation for the line of sight ray is: ; Solve the following system of inequalities: ; in, Here, represents the distance extended along the line of sight from the pupil; if it exists... If the solution is not found, then determine whether the line of sight intersects with the point of interest; if not, then... If the solution is correct, then it is determined that the line of sight does not intersect with the point of interest; S43. Calculate the time tourists spend looking at candidate focal points of interest. Interest scores are calculated by combining the confidence level of tourists’ focus on emotional tags; Calculate tourists' interest scores for candidate focus areas of interest. as follows: ; in, Based on the base stay time, To focus on the confidence level of sentiment labels, , all These are the weighting coefficients. The candidate focus of interest is then determined as the current focus of interest. S44. Recalculate the interest score every 500ms. If the score of the new candidate interest focus exceeds the current interest focus by more than 0.2, update the interest focus label; otherwise, keep the current interest focus label. S45. Output the tourist's current interest focus tag (such as specific cultural relic A, architectural detail B) and interest score, and record the switching frequency of interest focus tags (to help determine the tourist's exploration pace).

[0025] S5. Based on tourists' real-time emotional state and focus of interest, dynamically select and organize information from the knowledge graph to generate personalized narrative text that is context-appropriate, of suitable length, and emotionally resonant. It employs a three-layer architecture: knowledge graph support, narrative logic planning, and natural language text generation. Each layer collaborates to achieve personalized language text generation, as detailed below: S51. Define a knowledge graph for the tourism sector, with attractions and points of interest as core nodes, construct a multi-relationship, multi-level structured knowledge network, including entity nodes, relationship types, attribute information, and provide knowledge graph storage and retrieval functions; Entity nodes: POI nodes (such as cultural relic A, building B), attribute nodes (such as age, material, historical background), story nodes (such as related anecdotes, historical events), and guide nodes (such as rest tips and tour suggestions); Relationship type: Has attributes (Artifact A - Has attributes - Era: Song Dynasty), Related stories (Architecture B - Related stories - Anecdotes C), Matching emotions (Tired - Matching content - Brief rest prompt). Attribute information: Each POI node is associated with multi-dimensional content materials, which are divided into detailed version (more than 500 words, including technical background and historical details), concise version (100-200 words, core information) and interesting version (50-100 words, anecdotes and interesting stories) according to granularity. Knowledge graph storage and retrieval: The knowledge graph is stored using a graph database (such as Neo4j), which supports fast retrieval based on node attributes and relationship types. The retrieval response time is ≤100ms, and dynamic updates (such as adding POI content materials) are also supported. S52. Emotion-interest driven narrative logic planning: constructing a narrative logic rule base and using dynamic programming algorithm to select the optimal narrative structure; Narrative Logic Rule Base: Predefines narrative strategies corresponding to different emotion-interest combinations, some of which are shown in Table 2; Table 2. Narrative Logic Rules Table;

[0026] Dynamic programming algorithm: Based on the narrative logic rule base, a greedy algorithm is used to dynamically select the optimal narrative path. The steps are as follows: Input: Emotional state label E, Interest focus label P, Knowledge graph G; a: Retrieves all content nodes (attributes, stories, etc.) in G that are associated with P; b: Filter content nodes that match E (e.g., if E = Curiosity, filter detailed attribute nodes and in-depth analysis story nodes); c: Calculate the adaptation score for each content node: ; in, For the relevance of content to interests, For the fit between content and emotion, , ; d: Sort by score in descending order, select the top 3 content nodes, and organize the narrative structure according to the logical order of core attributes → extended information → emotional resonance points; S53. It adopts a fine-tuning architecture based on a pre-trained language model (such as BERT+GPT-2), with the input being interest focus labels, sentiment state labels, and narrative structure prompt words, and the output being natural language text; Input prompt format: Generate guide text about {interest focus P}, the visitor's current emotion is {emotion E}, and it must follow the following structure: {structure S}, language style is {style F}, and the length is controlled within {length L} words; Model fine-tuning: Fine-tuning was performed based on a tourism guide corpus (containing 100,000+ scenic spot explanation texts of different styles), with optimization goals being text fluency, information accuracy, and sentiment matching. S54. The output natural language text is optimized by length verification (if it exceeds the preset length, the core information is automatically simplified) and grammar correction (using NLP grammar checking tools). Finally, a personalized narrative text that meets the requirements is output (length: 50-500 words, dynamically adjusted according to emotional state, such as ≤100 words when tired, ≥300 words when curious). The text also includes emotion guidance markers (such as [excitement], [relaxation]) to guide subsequent speech synthesis.

[0027] S6. Convert the personalized narrative text generated in S5 into speech for broadcast. The specific process of converting the generated personalized narrative text into speech with corresponding emotional color and artistic expression for broadcasting is as follows: S61. Design an emotion mapping and speech generation mechanism to map abstract emotion state labels into quantifiable acoustic parameters, driving the speech synthesis model to generate speech that conforms to emotional features. S62. Select the core acoustic parameters that affect the expression of emotion in speech, and use a multiple linear regression model to establish the mapping from emotional state labels to acoustic parameters; Acoustic parameter definition: Select the core acoustic parameters that affect the emotional expression of speech, as shown in Table 3: Table 3. Definition of acoustic parameters;

[0028] The mapping process is as follows: S621. Suppose that the emotional state labels are converted into vectors through one-hot encoding. , , , , Corresponding to curiosity, focus, pleasure, and fatigue respectively, the acoustic parameter vectors are: The mapping relationship is then: ; in, The weight matrix is ​​5×4, representing the influence weight of each emotional state on each acoustic parameter. It is a 5-dimensional bias vector. This is the random error term; S622. Using an emotional speech corpus (containing 1000 different emotional speech recordings from 10 professional broadcasters, each emotion lasting 3-5 seconds), solve the problem using the least squares method. and The formula for minimizing the mean square error (MSE) between the predicted and actual acoustic parameters is as follows: ; in, For the first The true acoustic parameters of each sample For the first The sentiment vector of each sample; S63. The speech synthesis model adopts an end-to-end Transformer-TTS architecture, introduces emotional acoustic parameter constraints in the acoustic decoder, and uses the mapped acoustic parameters as the conditional input of the acoustic decoder so that the generated Mel spectrum meets the requirements of emotional features. Basic model architecture: It adopts an end-to-end Transformer-TTS architecture, which includes a text encoder (converting text into semantic vectors), an acoustic decoder (converting semantic vectors into Mel spectra), and a vocoder (converting Mel spectra into speech waveforms). Emotion-driven optimization: Emotional acoustic parameter constraints are introduced into the acoustic decoder, and the acoustic parameters output by the mapping model are used as the conditional input of the decoder so that the generated Mel spectrum meets the requirements of emotional features; Voice adaptation: It has a built-in library of multiple basic voices (young female voice, young male voice, middle-aged male voice, etc.) and dynamically selects the appropriate voice based on emotional tags (e.g., curiosity → young female voice, nostalgia → middle-aged male voice). It also supports custom voice selection by guests. S64. Perform noise reduction processing on the generated speech waveform (use spectral subtraction to remove environmental noise) and volume normalization (make the average volume value uniform to -6dB) to ensure speech clarity and auditory comfort. S65. Outputs voice through multiple terminals and updates acoustic parameters in real time after the tourist's emotional state changes, reflecting the emotional switch in the next voice segment to ensure real-time matching between voice and tourist's state; Terminal compatibility: Supports various terminals such as tourists' mobile phones, Bluetooth headsets, and tour guide equipment provided by attractions, and transmits audio streams via Bluetooth or Wi-Fi; Real-time adjustment: If the tourist's emotional state changes (such as from curiosity to fatigue), the engine updates the acoustic parameters in real time and reflects the emotional switch in the next speech segment to ensure that the speech matches the tourist's state in real time.

[0029] like Figure 2 As shown, the personalized voice guide generation system based on tourists' emotional states includes a multi-dimensional data acquisition module, a preprocessing module, an emotion recognition module, an interest focus calculation module, a text generation module, and a voice conversion module. The multi-dimensional data acquisition module is responsible for facial data acquisition and physiological data acquisition; The preprocessing module is responsible for cleaning and synchronizing facial and physiological data, extracting features, and standardizing the data. The emotion recognition module is responsible for single-modal feature encoding, cross-modal feature fusion, and emotion classification, outputting the emotional state labels and confidence scores of tourists. The interest focus calculation module is responsible for determining the candidate interest focus of tourists by associating points of interest and calculating the interest score of the candidate interest focus to update the candidate interest focus; The text generation module is responsible for providing knowledge graph storage and retrieval, planning narrative logic to output natural language text, and then optimizing the natural language text to generate personalized narrative text. The speech conversion module is responsible for designing the emotion mapping and speech generation mechanism to generate speech that conforms to emotional characteristics.

[0030] Therefore, the present invention adopts the personalized voice guide generation method and system based on the emotional state of tourists, which accurately reflects the real-time emotional state and interest preferences of tourists through multi-dimensional data collection, constructs an emotion recognition model to calculate and output the emotional state of tourists, calculates the tourists' interest focus based on the tourists' visual modal features and points of interest (POIs), and generates personalized narrative text that is context-appropriate, of suitable length, and emotionally matched based on "emotion-interest" and converts it into voice broadcast. This can provide more considerate voice services for tourist areas, scenic spots and other service scenarios, attract more visitors, increase the number of tourists, drive the surrounding economic development, and attract corporate investment.

[0031] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the technical solutions of the present invention, and these modifications or equivalent substitutions cannot cause the modified technical solutions to deviate from the spirit and scope of the technical solutions of the present invention.

Claims

1. A method for generating personalized voice guides based on tourists' emotional states, characterized in that, Includes the following steps: S1. Collect multi-dimensional data reflecting tourists' real-time emotional state and interest preferences; S2. After preprocessing the multi-dimensional data collected in S1, visual modal features and physiological modal features are extracted. S3. Based on the visual modal features and physiological modal features of S2, construct and train an emotion recognition model to output the emotional state labels of tourists; S4. Calculate the tourist's interest focus tags based on the tourist's visual modal characteristics and points of interest; S5. Based on tourists' real-time emotional state and focus of interest, dynamically select and organize information from the knowledge graph to generate personalized narrative text that is context-appropriate, of suitable length, and emotionally resonant. S6. Convert the personalized narrative text generated in S5 into speech for broadcast.

2. The personalized voice guide generation method based on tourist emotional state according to claim 1, characterized in that: The devices in S1 that collect multi-dimensional data include high-definition cameras deployed in key points of interest areas of the attraction to collect visitors' facial data, and lightweight wearable devices distributed to visitors upon entry to collect visitors' physiological data.

3. The personalized voice guide generation method based on tourist emotional state according to claim 2, characterized in that: The specific process of S2 is as follows: S21. Remove outliers to complete data cleaning and use interpolation to fill in temporary data gaps; S22. Based on timestamps, facial data and physiological data are aligned to complete data synchronization and construct a unified multimodal data time series; S23. Extract visual modal features. Extract the coordinates of key facial feature points using a multi-task convolutional neural network (MTCNN) and calculate the geometric features between the coordinates of key facial feature points. The gaze focus area is calculated based on the spatial intersection of the gaze ray and the coordinates of the scene interest points. S24. Extract physiological modal features by performing frequency domain analysis and smoothing on physiological data to extract peak and mean features; S25. The extracted visual modal features and physiological modal features are normalized using the Z-score normalization method. The specific calculation formula is as follows: ； in, For the first i The first time point j 1 eigenvalue, For the first j The mean of each feature, For the first j The standard deviation of each feature.

4. The personalized voice guide generation method based on tourist emotional state according to claim 3, characterized in that: The S3 emotion recognition model adopts a three-level architecture: a single-modal feature encoding layer, a cross-modal feature fusion layer, and an emotion classification layer, as detailed below: S31. Design a single-modal feature coding layer that includes visual modality coding and physiological modality coding; Visual modality encoding uses a convolutional neural network (CNN) architecture. The input is a temporal sequence of key facial feature points. Local spatiotemporal features are extracted through three convolutional layers, and temporal dependencies are captured through an LSTM layer to output a visual modality feature vector. Physiological modality coding: A multilayer perceptron (MLP) architecture is adopted. The input is the standardized physiological modality features, which are mapped through three fully connected layers to output physiological modality feature vectors. S32. Design a cross-modal feature fusion layer, using an attention mechanism to achieve adaptive fusion of multimodal features, and calculate the association weights between visual modal feature vectors and physiological modal feature vectors, as shown in the following formula: ； ； in, For visual modality feature vectors, This is a feature vector representing physiological modalities. Attention weights for visual features This is the weight matrix. For bias terms, The square root of the visual feature dimension is used to dynamically adjust the contribution of feature vectors of different modalities. S33. Design the sentiment classification layer, using an MLP architecture, with the input being a fused feature vector. Through two fully connected layers and a Dropout layer, the probability distribution of various emotions is finally output through the Softmax activation function, as shown in the following formula: ； in, For the emotion category, among which =4 and When =1 corresponds to curiosity. =2 corresponds to focus. =3 corresponds to pleasure. =4 corresponds to fatigue. For the first Class weight matrix, For the first Class bias terms, The weight matrix is ​​responsible for integrating the fused multimodal features. Mapped to the Emotional type for The accompanying bias term is used to adjust the decision boundary of sentiment classification, taking the category with the highest probability as the final sentiment state label; S34. Output the tourist's emotional state label and corresponding confidence level.

5. The personalized voice guide generation method based on tourist emotional state according to claim 4, characterized in that: The specific training process of the emotion recognition model in S3 is as follows: The training dataset combines a public sentiment dataset with a custom tourist attraction dataset. The custom tourist attraction dataset contains multimodal data from 1,000 tourists of different ages and genders, labeled with four sentiment tags. The cross-entropy loss function is used, and the formula is as follows: ； in, The number of training samples, For the first The true label of each sample For the first The predicted value for the nth sample is... The probability of similar emotions; The Adam optimizer was used, with an initial learning rate of 0.001, which was reduced to 0.9 every 10 epochs. The training iterations were performed for 100 epochs until the loss function converged.

6. The personalized voice guide generation method based on tourist emotional state according to claim 5, characterized in that: The specific process for calculating the focus of interest in S4 is as follows: S41. Calculate the spatial equation of the line-of-sight ray based on pupil position data from visual modal features acquired by the camera: ； in, , , Associate each point of interest with a unique range of spatial coordinates. 3D coordinates of the visitor's pupils The direction vector of the line of sight is calculated by connecting the pupil and the feature point at the corner of the eye. S42. Perform interest point association judgment. Determine whether the line of sight intersects with the spatial coordinate range of a certain interest point. If they intersect, record the interest point as a candidate interest focus. If they do not intersect, do not record it. To determine whether a line of sight intersects the spatial coordinate range of a point of interest, a spatial geometric intersection algorithm is used, as shown in the following formula: Let the spatial extent of the point of interest be a cuboid: , , The parametric equation for the line of sight ray is: ； Solve the following system of inequalities: ； in, Here, represents the distance extended along the line of sight from the pupil; if it exists... If the solution is not found, then determine whether the line of sight intersects with the point of interest; if not, then... If the solution is correct, then it is determined that the line of sight does not intersect with the point of interest; S43. Calculate the time tourists spend looking at candidate focal points of interest. Interest scores are calculated by combining the confidence level of tourists’ focus on emotional tags; Calculate tourists' interest scores for candidate focus areas of interest. as follows: ； in, Based on the base stay time, To focus on the confidence level of sentiment labels, , All are weighting coefficients. The candidate focus of interest is then determined as the current focus of interest. S44. Recalculate the interest score every 500ms. If the score of the new candidate interest focus exceeds the current interest focus by more than 0.2, update the interest focus label; otherwise, keep the current interest focus label. S45. Output the tourist's current interest focus tags and interest score, and record the switching frequency of interest focus tags.

7. The personalized voice guide generation method based on tourist emotional state according to claim 6, characterized in that: S5 employs a three-layer architecture: knowledge graph support, narrative logic planning, and natural language text generation. Each layer collaborates to achieve personalized language text generation, as detailed below: S51. Define a knowledge graph for the tourism sector, with attractions and points of interest as core nodes, construct a multi-relationship, multi-level structured knowledge network, including entity nodes, relationship types, attribute information, and provide knowledge graph storage and retrieval functions; S52. Emotion-interest driven narrative logic planning: constructing a narrative logic rule base and using dynamic programming algorithm to select the optimal narrative structure; S53. It adopts a fine-tuning architecture based on a pre-trained language model. The input is interest focus labels, sentiment state labels, and narrative structure prompts, and the output is natural language text. S54. Perform length verification and syntax correction optimization on the output natural language text, and finally output personalized narrative text that meets the requirements.

8. The personalized voice guide generation method based on tourist emotional state according to claim 7, characterized in that: The specific process by which S6 transforms the generated personalized narrative text into speech with corresponding emotional color and artistic expression for broadcasting is as follows: S61. Design an emotion mapping and speech generation mechanism to map abstract emotion state labels into quantifiable acoustic parameters, driving the speech synthesis model to generate speech that conforms to emotional features. S62. Select the core acoustic parameters that affect the expression of emotion in speech, and use a multiple linear regression model to establish the mapping from emotional state labels to acoustic parameters; S63. The speech synthesis model adopts an end-to-end Transformer-TTS architecture, introduces emotional acoustic parameter constraints in the acoustic decoder, and uses the mapped acoustic parameters as the conditional input of the acoustic decoder so that the generated Mel spectrum meets the requirements of emotional features. S64. Perform noise reduction and volume normalization on the generated speech waveform; S65. Outputs voice through multiple terminals and updates acoustic parameters in real time after the tourist's emotional state changes, reflecting the emotional switch in the next voice segment to ensure real-time matching between voice and tourist's state.

9. The personalized voice guide generation method based on tourist emotional state according to claim 8, characterized in that: The mapping process for S62 is as follows: S621. Suppose that the emotional state labels are converted into vectors through one-hot encoding. , , , , Corresponding to curiosity, focus, pleasure, and fatigue respectively, the acoustic parameter vectors are: The mapping relationship is then: ； in, The weight matrix is ​​5×4, representing the influence weight of each emotional state on each acoustic parameter. It is a 5-dimensional bias vector. This is the random error term; S622. Using an emotional speech corpus, solve the problem using the least squares method. and The formula for minimizing the mean square error (MSE) between the predicted and actual acoustic parameters is as follows: ； in, For the first The true acoustic parameters of each sample For the first The sentiment vector of each sample.

10. A personalized voice guide generation system based on tourists' emotional states, characterized in that: The system, which uses any one of the personalized voice guide generation methods based on tourist emotional states as described in claims 1-9, includes a multi-dimensional data acquisition module, a preprocessing module, an emotion recognition module, an interest focus calculation module, a text generation module, and a voice conversion module. The multi-dimensional data acquisition module is responsible for facial data acquisition and physiological data acquisition; The preprocessing module is responsible for cleaning and synchronizing facial and physiological data, extracting features, and standardizing the data. The emotion recognition module is responsible for single-modal feature encoding, cross-modal feature fusion, and emotion classification, outputting the emotional state labels and confidence scores of tourists. The interest focus calculation module is responsible for determining the candidate interest focus of tourists by associating points of interest and calculating the interest score of the candidate interest focus to update the candidate interest focus; The text generation module is responsible for providing knowledge graph storage and retrieval, planning narrative logic to output natural language text, and then optimizing the natural language text to generate personalized narrative text. The speech conversion module is responsible for designing the emotion mapping and speech generation mechanism to generate speech that conforms to emotional characteristics.

Images