Intelligent travel customer service response system and method based on multi-modal emotion recognition

Abstract

The invention discloses a text travel intelligent customer service response system and method based on multi-modal emotion recognition, and relates to the technical field of artificial intelligence and man-machine interaction, and the method comprises the steps: starting second modal perception when a first modal cognitive load index reaches a threshold value and the matching degree between a load interval of the first modal cognitive load index and a preset heart flow interval is lower than the threshold value; integrating the two modal indexes to determine a comprehensive cognitive load state and a change trend; if in an overload interval, calling a fuzzy control logic dynamic mapping algorithm to generate an optimal narrative strategy vector containing parameters such as information density and presentation mode, retrieving a micro-narrative unit library and serially connecting to generate personalized content and multi-channel presentation, and adjusting a strategy according to a comprehensive state, such as a heart flow interval maintenance strategy and fine adjustment of the parameters. The interaction depth of the idle interval is improved, accurate adaptation of the text travel service is realized through multi-modal fusion and dynamic strategy adjustment, and the user experience is improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and human-computer interaction technology, and more specifically, to a smart customer service response system and method for cultural tourism based on multimodal emotion recognition. Background Technology

[0002] In the process of upgrading from cultural and tourism integration to in-depth experience, smart customer service has transformed from functional response to emotional adaptation. However, the current service system still has a technological gap in dynamically perceiving the user's cognitive state. Existing solutions mostly focus on single-modal emotion classification, simplifying emotions into discrete labels, but ignoring the coupling relationship between cognitive load and emotional experience in cultural and tourism scenarios. In scenarios such as historical exhibition halls and natural scenic spots, the efficiency of tourists' information reception is directly constrained by the dynamic changes in visual focus intensity, interactive operation complexity, and environmental interference.

[0003] While the application of flow theory in cultural and tourism experience design has garnered attention, traditional systems lack precise anchoring of cognitive load ranges. This often leads to flow state interruptions or cognitive overload due to a mismatch between information density and user processing capacity. At the multimodal data fusion level, existing technologies either rely solely on a single dimension of vision or speech, or treat behavioral interaction, environmental soundprints, and other modal data with equal weight, failing to reflect the differentiated value of different modalities in representing cognitive states. On the content supply side, most systems use fixed narrative templates. Even with the introduction of personalized recommendations, a dynamic mapping mechanism between strategy parameters and cognitive states has not been established. The invocation of micro-narrative units lacks refined control over information density and interaction depth. Furthermore, service adjustments are mostly static responses, lacking predictive optimization based on load change trends, making it difficult to upgrade from passive adaptation to proactive guidance. This hinders the improvement of immersion and satisfaction in cultural and tourism services.

[0004] Therefore, existing technologies suffer from problems such as one-sided single-modal perception, inaccurate load judgment, and poor adaptability between narrative strategies and cognitive states. Summary of the Invention

[0005] In order to overcome the problems of single-modal perception bias, inaccurate load judgment, and poor adaptability between narrative strategies and cognitive states in existing technologies, this invention discloses a smart customer service response system and method for cultural tourism based on multimodal emotion recognition, which can effectively solve the above-mentioned technical problems.

[0006] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows: A smart customer service response method for cultural tourism based on multimodal emotion recognition is applied to a smart cultural tourism service system. The method includes: When the first modality cognitive load index reaches a first threshold, and the matching degree between the load interval to which the first modality cognitive load index belongs and the preset flow interval is less than a preset matching threshold, the second modality cognitive load sensing is activated. Based on the first modal cognitive load index and the second modal cognitive load index, the user's overall cognitive load status and load change trend are determined; When the overall cognitive load is in the overload range, the dynamic mapping algorithm based on fuzzy control logic is invoked to generate the optimal narrative strategy vector. Based on the optimal narrative strategy vector, matching micro-narrative units are retrieved from the micro-narrative unit library and concatenated to generate personalized service content, which is then presented to the user through multi-channel adaptive methods. The optimal narrative strategy vector includes parameters such as information density, presentation mode, emotional tone, and interaction depth, and the micro-narrative unit library contains structured cultural and tourism knowledge units with pre-labeled narrative parameter tags.

[0007] Preferably, the method further includes: when the matching degree between the load interval corresponding to the first modality cognitive load index and the preset flow interval is not less than a preset matching threshold, maintaining the current narrative strategy and pushing optional in-depth expansion service content.

[0008] Preferably, before invoking the dynamic mapping algorithm based on fuzzy control logic to generate the optimal narrative strategy vector when the comprehensive cognitive load state is in the overload range, the method further includes: determining the load range to which the comprehensive cognitive load state belongs; When the overall cognitive load is in the flow interval, maintain the current narrative strategy vector and periodically fine-tune the information density parameter; when it is in the idle interval, increase the interaction depth parameter and push challenging question-type service content.

[0009] Preferably, determining the user's overall cognitive load status and load change trend includes: assigning weights to the first modality cognitive load index and the second modality cognitive load index, wherein the weight of the behavioral interaction modality is higher than that of the visual posture modality, and the weight of the visual posture modality is higher than that of the environmental soundprint modality. A comprehensive cognitive load index is generated based on the weighted fusion results, and the load change slope of several consecutive sampling points is calculated through time series analysis to determine the load change trend.

[0010] Preferably, the method further includes: monitoring changes in the cognitive load index in real time during the presentation of personalized service content; When the load change trend shows that the rate of decline exceeds the preset threshold, the information density parameter is gradually reduced back to the baseline value.

[0011] Preferably, the first modality includes a visual pose modality, and the second modality includes a behavioral interaction modality and an environmental soundprint modality; The first modal cognitive load index is generated by calculating head dwell time and gaze drift frequency, while the second modal cognitive load index is generated by fusing device operation behavior characteristics and environmental sound entropy values.

[0012] Preferably, the generation of personalized service content includes: retrieving the combination of micro-narrative units with the highest matching degree from the micro-narrative unit library based on the parameter tags in the optimal narrative strategy vector; By using retrieval-enhanced generation technology, the micro-narrative units are linked into coherent content using transition language, wherein the style of the transition language is consistent with the emotional tone parameters.

[0013] Preferably, the method further includes: periodically collecting narrative strategy adjustment data and corresponding load change curves under different load intervals, and optimizing the fuzzy rule base in the dynamic mapping algorithm through cluster analysis.

[0014] Preferably, the method further includes: when the user input contains keywords for emergency help, skipping the cognitive load judgment process, directly switching to the highest priority human customer service channel, and simultaneously pushing emergency guidance content for the current scenario.

[0015] Preferably, the cultural tourism intelligent customer service response system based on multimodal emotion recognition includes: The modal perception module, when the first modal cognitive load index reaches a first threshold, activates the second modal cognitive load perception when the matching degree between the load interval corresponding to the first modal cognitive load index and the preset flow interval is less than a preset matching threshold. The load analysis module determines the user's overall cognitive load status and load change trend based on the first modal cognitive load index and the second modal cognitive load index. The content generation module, when the overall cognitive load is in the overload range, calls a dynamic mapping algorithm based on fuzzy control logic to generate an optimal narrative strategy vector. Based on the optimal narrative strategy vector, it retrieves matching micro-narrative units from the micro-narrative unit library and concatenates them to generate personalized service content, which is then presented to the user through multi-channel adaptive processing. The optimal narrative strategy vector includes parameters such as information density, presentation mode, emotional tone, and interaction depth, and the micro-narrative unit library contains structured cultural and tourism knowledge units with pre-labeled narrative parameter tags.

[0016] Compared with existing technologies, the beneficial effects of this invention are as follows: This method achieves significant progressive benefits through end-to-end technological innovation in perception-analysis-adaptation-optimization: Addressing the problem of one-sided perception in a single modality, a primary and secondary modality hierarchical activation mechanism is adopted. Visual posture modality is used as the first perceptual dimension (calculated by indices based on head dwell time and gaze drift frequency) for initial judgment. Only when it reaches a threshold and its matching degree with the flow interval is insufficient is the second modality, composed of behavioral interaction and environmental soundprints (calculated by fusing device operation features and environmental sound entropy values), activated. The complementary verification of multimodal data avoids misjudgments of cognitive states caused by missing information in a single dimension. For example, in scenic area navigation, it can capture both the user's wandering gaze due to information overload and, combined with repeated screen swiping operations and noisy environmental soundprints, comprehensively restore the real cognitive scenario. Addressing the problem of inaccurate load judgment, a weighted fusion of multimodal indices is achieved through differentiated weight allocation (behavioral interaction > visual posture > environmental soundprints) to generate a more representative comprehensive index. By combining time-series analysis to calculate the slope of continuous sampling points and predict changing trends, this approach avoids the weakening of key information caused by equal weighting and solves the problem that static judgment cannot capture dynamic changes. For example, it can accurately distinguish between different load states represented by tourists' instantaneous gaze shifts and continuous operational sluggishness. To address the problem of poor narrative strategy adaptability, a state-strategy dynamic mapping system is constructed. Based on fuzzy control logic algorithms, load states such as overload, flow, and idle are transformed into optimal narrative strategy vectors containing parameters such as information density and emotional tone. Micro-narrative units are matched through retrieval enhancement generation technology and connected with adaptive transitions. At the same time, a real-time feedback mechanism is established—density is reduced when overloaded, parameters are fine-tuned when in a flow state, and interaction depth is increased when idle. After the load decreases to a certain level, parameters are gradually reverted to form a closed-loop adaptation. In addition, the fuzzy rule base is optimized regularly through cluster analysis to continuously improve the accuracy of strategy matching. This changes the rigidity of traditional fixed template services and achieves dynamic and accurate matching between cultural tourism services and users' cognitive states, thereby improving immersion and satisfaction. Attached Figure Description

[0017] To more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely exemplary. For those skilled in the art, other embodiments can be derived from the provided drawings without creative effort.

[0018] Figure 1 This is a diagram illustrating the steps of the method of the present invention; Figure 2 This is a system structure diagram of the present invention. Detailed Implementation

[0019] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent. To better illustrate this embodiment, some parts in the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions; It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings.

[0020] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments. Example 1

[0021] Please see Figure 1 A smart customer service response method for cultural tourism based on multimodal emotion recognition is applied to a smart cultural tourism service system. The method includes: When the first modality cognitive load index reaches a first threshold, and the matching degree between the load interval to which the first modality cognitive load index belongs and the preset flow interval is less than a preset matching threshold, the second modality cognitive load sensing is activated. Based on the first modal cognitive load index and the second modal cognitive load index, the user's overall cognitive load status and load change trend are determined; When the overall cognitive load is in the overload range, the dynamic mapping algorithm based on fuzzy control logic is invoked to generate the optimal narrative strategy vector. Based on the optimal narrative strategy vector, matching micro-narrative units are retrieved from the micro-narrative unit library and concatenated to generate personalized service content, which is then presented to the user through multi-channel adaptive methods. The optimal narrative strategy vector includes parameters such as information density, presentation mode, emotional tone, and interaction depth, and the micro-narrative unit library contains structured cultural and tourism knowledge units with pre-labeled narrative parameter tags.

[0022] The method further includes: when the matching degree between the load interval corresponding to the first modality cognitive load index and the preset flow interval is not less than the preset matching threshold, maintaining the current narrative strategy and pushing optional in-depth expansion service content.

[0023] Before invoking the dynamic mapping algorithm based on fuzzy control logic to generate the optimal narrative strategy vector when the overall cognitive load state is in the overload range, the method further includes: determining the load range to which the overall cognitive load state belongs; When the overall cognitive load is in the flow interval, maintain the current narrative strategy vector and periodically fine-tune the information density parameter; when it is in the idle interval, increase the interaction depth parameter and push challenging question-type service content.

[0024] The determination of the user's overall cognitive load status and load change trend includes: weighting the first modality cognitive load index and the second modality cognitive load index, wherein the weight of the behavioral interaction modality is higher than that of the visual posture modality, and the weight of the visual posture modality is higher than that of the environmental soundprint modality. A comprehensive cognitive load index is generated based on the weighted fusion results, and the load change slope of several consecutive sampling points is calculated through time series analysis to determine the load change trend.

[0025] The method further includes: real-time monitoring of changes in the cognitive load index during the presentation of personalized service content; When the load change trend shows that the rate of decline exceeds the preset threshold, the information density parameter is gradually reduced back to the baseline value.

[0026] The first modality includes a visual pose modality, and the second modality includes a behavioral interaction modality and an environmental voiceprint modality; The first modal cognitive load index is generated by calculating head dwell time and gaze drift frequency, while the second modal cognitive load index is generated by fusing device operation behavior characteristics and environmental sound entropy values.

[0027] The generation of personalized service content includes: retrieving the combination of micro-narrative units with the highest matching degree from the micro-narrative unit library based on the parameter tags in the optimal narrative strategy vector; By using retrieval-enhanced generation technology, the micro-narrative units are linked into coherent content using transition language, wherein the style of the transition language is consistent with the emotional tone parameters.

[0028] The method also includes: periodically collecting narrative strategy adjustment data and corresponding load change curves under different load ranges, and optimizing the fuzzy rule base in the dynamic mapping algorithm through cluster analysis.

[0029] The method further includes: when the user input contains keywords for emergency help, skipping the cognitive load judgment process, directly switching to the highest priority human customer service channel, and simultaneously pushing emergency guidance content for the current scenario.

[0030] In practice, in cultural and tourism scenic spots or related service venues, cameras installed in key locations, such as scenic spot entrances, popular attractions, and visitor service centers, collect visual posture data of tourists. These cameras have high resolution and low latency characteristics, and can clearly capture information such as tourists' head posture and gaze direction. For example, at the entrance of a scenic spot, after tourists enter the camera's field of view, the system begins to capture their head posture in real time. Through computer vision algorithms, it analyzes the tourists' head posture parameters such as pitch angle and yaw angle, as well as indicators such as gaze dwell time and gaze drift frequency.

[0031] The specific calculation formula is: Visual posture cognitive load index = w1 × (head dwell time / average dwell time baseline) + w2 × (gaze drift frequency / average drift frequency baseline) w1 and w2 are weighting coefficients, set to 0.6 and 0.4 respectively based on experimental data and experience. The average dwell time benchmark and the average drift frequency benchmark are obtained through statistical analysis of a large amount of normal tourist browsing behavior data and are used as reference standards to measure the current tourist visual posture cognitive load.

[0032] The system pre-sets a first threshold for the first modality cognitive load index. When the calculated visual posture cognitive load index reaches this threshold, the subsequent judgment process is triggered.

[0033] The system calculates the matching degree between the current visual posture cognitive load index corresponding to the load interval and the preset flow interval. The flow interval is set according to psychological principles and combined with the ideal experience state of tourists in cultural and tourism scenarios (neither too relaxed to cause boredom, nor too tense to cause anxiety). For example, the cognitive load index range corresponding to the flow interval is [0.4, 0.6].

[0034] The matching degree can be calculated by interval overlap. If the proportion of the overlapping part to the cardiac flow interval is less than the preset matching threshold (e.g., 0.3), the matching degree is determined to be less than the preset matching threshold, and then the second modality cognitive load perception is activated.

[0035] The system monitors the interaction between tourists and smart devices within the scenic area, such as smart guide screens and interactive experience devices. Through the built-in sensors and log recording functions of these devices, it acquires the characteristics of tourists' operational behavior, such as operation frequency, operation duration, and operation path. For example, when tourists search for attraction information on a smart guide screen, the system records the frequency of their clicks on the search button, the time spent on different pages, and the order in which they browse attraction introductions.

[0036] Based on this data, the cognitive load index of behavioral interaction modality is calculated using the following formula: Cognitive Load Index for Behavioral Interaction = a1 × (Operation Frequency / Average Operation Frequency Baseline) + a2 × (Operation Duration / Average Operation Duration Baseline) + a3 × (Operation Path Complexity / Average Path Complexity Baseline) Where a1, a2, and a3 are weighting coefficients, with values ​​of 0.4, 0.3, and 0.3 respectively. Each benchmark value is also derived from statistical analysis of a large amount of normal tourist behavior data.

[0037] Environmental sound sensors are deployed within the scenic area to collect sound signals from the environment in which tourists are located. These sensors can distinguish between different types of sound sources, such as wind, birdsong, tourists talking, and equipment operation.

[0038] The entropy value of ambient sound is calculated using a voiceprint analysis algorithm. The entropy value reflects the complexity and disorder of ambient sound. A higher entropy value indicates that the ambient sound is relatively noisy, which may increase the cognitive load of tourists.

[0039] The formula for calculating the environmental soundprint modal cognitive load index is: Environmental soundprint cognitive load index = b × (environmental sound entropy value / average sound entropy value baseline) Where b is the weighting coefficient, which is 0.5, and the average sound entropy benchmark is obtained based on the statistical analysis of environmental sound in different areas during normal operation of the scenic area.

[0040] According to the technical solution, the weight of the behavioral interaction modality is higher than that of the visual posture modality, and the weight of the visual posture modality is higher than that of the ambient sound signature modality. The specific weight allocation is as follows: behavioral interaction modality weight 0.5, visual posture modality weight 0.3, and ambient sound signature modality weight 0.2.

[0041] The cognitive load indices of the first modality (visual posture modality) and the second modality (behavioral interaction modality and environmental voiceprint modality) are weighted and fused to generate a comprehensive cognitive load index: Comprehensive cognitive load index = 0.3 × visual posture cognitive load index + 0.5 × behavioral interaction cognitive load index + 0.2 × environmental soundprint cognitive load index A time-series analysis of the comprehensive cognitive load index was performed. Several consecutive sampling points were selected, such as sampling once every 30 seconds. Ten consecutive sampling points were selected, and the slope of load change between adjacent sampling points was calculated.

[0042] The formula for calculating the slope of load change is: The slope of load change = (comprehensive cognitive load index i − comprehensive cognitive load index i−1) / Δt, where Δt is the sampling time interval, which is 30 seconds in this case.

[0043] The load change trend is determined by the sign and magnitude of the slopes of multiple consecutive load changes. If most slopes are positive and the values ​​are large, it indicates that the cognitive load is on an upward trend. If most slopes are negative and the values ​​are large, it indicates that the cognitive load is on a downward trend. If the slopes are close to zero or alternate between positive and negative and the values ​​are small, it indicates that the cognitive load is relatively stable.

[0044] When the overall cognitive load is in the overload range, such as when the overall cognitive load index is greater than 0.8, the subsequent content generation module is triggered to call the dynamic mapping algorithm based on fuzzy control logic to generate the optimal narrative strategy vector.

[0045] If the overall cognitive load is in the flow zone, such as when the overall cognitive load index is in the range of [0.4, 0.6], maintain the current narrative strategy vector and periodically fine-tune the information density parameter. For example, fine-tune the information density parameter every 5 minutes. The fine-tuning range is determined according to the load change trend. If the load has a slight upward trend, the information density is appropriately reduced; if the load has a slight downward trend, the information density is appropriately increased, but the adjustment range does not exceed 10%.

[0046] When the overall cognitive load is in the idle range, such as when the overall cognitive load index is less than 0.2, increase the interaction depth parameter and push challenging question-based service content, such as pushing some interesting quizzes about the history and culture of the scenic spot to tourists, or recommending some hidden attraction information that requires tourists to participate in the interaction to unlock, so as to stimulate tourists' interest and increase their cognitive load to a reasonable level.

[0047] When the overall cognitive load is in the overload range, the content generation module calls a dynamic mapping algorithm based on fuzzy control logic. This algorithm pre-builds a fuzzy rule base and generates the optimal narrative strategy vector through fuzzy inference based on input parameters such as the overall cognitive load index, the load change trend, and the cognitive load index of each modality. For example, a rule in the fuzzy rule base can be: if the overall cognitive load index is high and the load change trend is rapidly increasing, and the cognitive load index of the behavioral interaction modality is high, then the information density parameter is low, the presentation mode parameter is concise text and graphics, the emotional tone parameter is soothing, and the interaction depth parameter is low.

[0048] The algorithm matches the input parameters with the rules in the fuzzy rule base and calculates the specific values ​​of each parameter in the optimal narrative strategy vector through fuzzy inference. For example, the information density parameter may be 0.2 (ranging from 0 to 1, with lower values ​​indicating lower information density), the presentation mode parameter may be 1 (ranging from 0 to 2, with 0 representing plain text, 1 representing simple text and images, and 2 representing multimedia video), the emotional tone parameter may be 0.3 (ranging from 0 to 1, with lower values ​​indicating a more soothing emotional tone), and the interaction depth parameter may be 0.1 (ranging from 0 to 1, with lower values ​​indicating a shallower interaction depth).

[0049] Based on the parameter labels in the generated optimal narrative strategy vector, the system retrieves the combination of micro-narrative units with the highest matching degree from the micro-narrative unit library. The micro-narrative unit library contains a large number of structured cultural and tourism knowledge units with pre-labeled narrative parameter labels. Each micro-narrative unit has corresponding parameter labels such as information density, presentation mode, emotional tone, and interaction depth. For example, the optimal narrative strategy vector indicates low information density, a concise text and image presentation mode, and a soothing emotional tone. The system then retrieves micro-narrative units from the micro-narrative unit library that match these parameter labels, such as a short text introducing the historical background of a scenic spot in the area, accompanied by a beautiful picture of the scenic spot, with a soothing and concise text style.

[0050] By using search-enhanced generation technology, the retrieved micro-narrative units are linked into coherent content using transition language. The style of the transition language is consistent with the emotional tone parameters. For example, under a soothing emotional tone, the transition language can use soft transition words, such as "Next, let us slowly walk into..." etc., so that the entire service content presents a natural and smooth presentation.

[0051] The resulting personalized service content can be presented to users adaptively through multiple channels. For example, for visual interaction devices (such as smart guide screens), it can be displayed in a concise graphic and text format; for voice interaction devices (such as smart voice guides), the text content can be converted into a soothing voice broadcast; for mobile terminals (such as cultural tourism apps on tourists' phones), the content layout and presentation can be adaptively adjusted according to the user's device screen size and interaction habits, such as displaying it in a graphic and text format on large screens and in a concise text and key image format on small screens.

[0052] During the presentation of personalized service content, the modal perception module continuously collects data such as visual posture, behavioral interaction, and environmental soundprints, and calculates the cognitive load index in real time.

[0053] When the load change trend shows that the rate of decline exceeds the preset threshold, such as a decrease of more than 0.1 per minute, the content generation module gradually reverts the information density parameter back to the baseline value. For example, if the current information density parameter is 0.2 and the baseline value is 0.5, the system will revert once every 30 seconds, each time by 0.1, until it reverts to the baseline value. This ensures that the cognitive load of tourists is not too low and is maintained within a reasonable range, thus guaranteeing the effective delivery of service content and a good experience for tourists.

[0054] Regularly collect narrative strategy adjustment data and corresponding load change curves under different load ranges. For example, extract data from the system once a quarter, including the comprehensive cognitive load index before and after each narrative strategy adjustment, the change curves of the cognitive load index of each modality, and the corresponding optimal narrative strategy vector parameter values.

[0055] By processing the collected data through cluster analysis, the fuzzy rule base in the dynamic mapping algorithm is optimized. Cluster analysis can discover the patterns and regularities of changes in tourists' cognitive load under different load ranges and narrative strategy adjustments. This allows for the adjustment, refinement, or addition of rules in the fuzzy rule base, improving the accuracy and adaptability of the algorithm and enabling it to better generate the optimal narrative strategy vector based on tourists' cognitive load status.

[0056] When the system detects that a user's input contains emergency help keywords such as "help," "lost," or "injured," it skips the cognitive load assessment process and directly switches to the highest-priority human customer service channel. At the same time, it pushes emergency guidance content based on the current scenario, such as the tourist's location and surrounding environment. For example, if a tourist is lost in the scenic area and enters the keyword "I'm lost," the system immediately switches to the human customer service channel. Human customer service personnel communicate with the tourist through video calls or other means, and at the same time, push the nearest scenic area map, exit directions, and other emergency information based on the tourist's location (via the tourist's mobile phone location or smart devices in the scenic area) to ensure that the tourist can get help quickly.

[0057] Through the above embodiments, the cultural tourism smart customer service response system can accurately perceive the cognitive load of tourists based on multimodal emotion recognition, dynamically generate personalized service content, effectively improve tourists' experience satisfaction in cultural tourism scenarios, and at the same time have good adaptability and emergency handling capabilities. Example 2

[0058] Please see Figure 2 A smart customer service response system for cultural tourism based on multimodal emotion recognition, the system comprising: The modal perception module, when the first modal cognitive load index reaches a first threshold, activates the second modal cognitive load perception when the matching degree between the load interval corresponding to the first modal cognitive load index and the preset flow interval is less than a preset matching threshold. The load analysis module determines the user's overall cognitive load status and load change trend based on the first modal cognitive load index and the second modal cognitive load index. The content generation module, when the overall cognitive load is in the overload range, calls a dynamic mapping algorithm based on fuzzy control logic to generate an optimal narrative strategy vector. Based on the optimal narrative strategy vector, it retrieves matching micro-narrative units from the micro-narrative unit library and concatenates them to generate personalized service content, which is then presented to the user through multi-channel adaptive processing. The optimal narrative strategy vector includes parameters such as information density, presentation mode, emotional tone, and interaction depth, and the micro-narrative unit library contains structured cultural and tourism knowledge units with pre-labeled narrative parameter tags.

[0059] This system mainly consists of three core modules: a modal perception module, a load analysis module, and a content generation module. The modal perception module is responsible for collecting users' multimodal data, including visual posture, behavioral interaction, and environmental soundprints, to assess users' cognitive load status. The load analysis module determines users' comprehensive cognitive load status and load change trends based on this data. When the comprehensive cognitive load status is in the overload range, the content generation module generates the optimal narrative strategy vector through a dynamic mapping algorithm of fuzzy control logic, retrieves matching micro-narrative units from the micro-narrative unit library, and concatenates them to generate personalized service content, which is then presented to users through multi-channel adaptive processing.

[0060] The modal perception module is the front end of the system, collecting and processing users' multimodal data in real time. First, the visual posture modality captures tourists' head posture and gaze direction through high-resolution cameras installed in the scenic area. These cameras can accurately record the tourists' head dwell time and gaze drift frequency, thereby initially assessing the tourists' cognitive load status. When the visual posture cognitive load index reaches the preset first threshold, the system will further determine the matching degree between the load interval corresponding to the index and the preset flow interval. If the matching degree is lower than the preset threshold, it indicates that the tourist may be in an uncomfortable cognitive load state. At this time, the system will start the second modality of cognitive load perception.

[0061] The second modality includes the behavioral interaction modality and the environmental soundprint modality. The behavioral interaction modality records tourists' operational behaviors through smart devices within the scenic area, such as operation frequency, operation duration, and operation path complexity. This data reflects the cognitive load of tourists when interacting with the devices. The environmental soundprint modality collects sound signals from the tourist's environment in real time through soundprint sensors deployed within the scenic area, including natural sounds, tourist conversations, and equipment operation sounds. By analyzing the complexity of the environmental sounds, i.e., the sound entropy value, the system can assess the impact of environmental sounds on tourists' cognitive load. The data from these two modalities will provide the system with a more comprehensive assessment of user cognitive load.

[0062] The load analysis module processes and analyzes data from the modality perception module. This module first performs weighted fusion of the cognitive load indices of the first and second modalities to generate a comprehensive cognitive load index. In terms of weight allocation, the behavioral interaction modality has the highest weight, followed by the visual posture modality, and finally the environmental soundprint modality. This weight allocation reflects the importance of different modalities in the impact on the user's cognitive load. Based on the weighted fusion results, the system can more accurately assess the user's comprehensive cognitive load status.

[0063] The system calculates the slope of load change at several consecutive sampling points through time-series analysis to determine the load change trend. This step enables the system to dynamically monitor changes in user cognitive load and adjust service content in a timely manner. In addition, the system presets different load intervals, including overload intervals, flow intervals, and idle intervals. Based on the comprehensive cognitive load index, the system can determine the user's current cognitive load status. If the user is in the flow interval, it means that their cognitive load is in an ideal state, and the system will maintain the current narrative strategy and periodically fine-tune the information density parameter to maintain this state. If the user is in the idle interval, the system will increase the interaction depth parameter and push challenging question-type service content to stimulate the visitor's interest and increase their cognitive load to a reasonable level.

[0064] The content generation module is the system's backend. It generates and pushes personalized service content based on the user's cognitive load. When the overall cognitive load is in the overload range, the system calls a dynamic mapping algorithm based on fuzzy control logic to generate an optimal narrative strategy vector. This vector includes four parameters: information density, presentation mode, emotional tone, and interaction depth. The information density parameter determines the amount of information in the service content; the presentation mode parameter determines the display format of the content, such as text, images, and videos; the emotional tone parameter determines the emotional tone of the content; and the interaction depth parameter determines the degree of interaction between the user and the system.

[0065] Based on the parameter labels in the optimal narrative strategy vector, the system retrieves the micro-narrative unit with the highest matching degree from the micro-narrative unit library. The micro-narrative unit library contains a large number of structured cultural and tourism knowledge units with pre-labeled narrative parameter labels. These units are the basis for the system to generate personalized service content. The system uses retrieval-enhanced generation technology to string together micro-narrative units into coherent service content through transition language. The style of the transition language is consistent with the emotional tone parameters to ensure the coherence and consistency of the content. Finally, the system adaptively adjusts the presentation of service content according to the user's device type, such as smart guide screen, mobile APP, voice guide, etc., to provide the best user experience.

[0066] Scene 1: Tourists at the entrance of the scenic area When tourists enter the scenic area, a high-resolution camera installed at the entrance begins to capture their head posture and gaze direction. The system records that the tourists' head stays on the ground for a short time and their gaze drifts frequently. The system calculates that the visual posture cognitive load index is 0.7, which reaches the preset first threshold. The system further determines that the load interval corresponding to this index matches the preset flow interval [0.4, 0.6] with a degree of 0.2, which is lower than the preset matching threshold of 0.3. Therefore, the system activates the second modality of cognitive load perception.

[0067] In the second mode, the system records tourists' operational behavior through the smart guide screen and finds that tourists operate frequently, for short periods of time, and with complex operation paths. The behavioral interaction cognitive load index is calculated to be 0.6. At the same time, the environmental soundprint sensor detects that the sound at the entrance of the scenic area is relatively noisy, and the environmental soundprint cognitive load index is calculated to be 0.5. The system performs weighted fusion of these data to generate a comprehensive cognitive load index of 0.65. Through time series analysis, the system finds that the load change trend is upward, and the comprehensive cognitive load index is close to the overload range.

[0068] To alleviate tourists' cognitive load, the system invokes a fuzzy control logic dynamic mapping algorithm to generate an optimal narrative strategy vector. The vector parameters include: low information density, concise text and image presentation mode, soothing emotional tone, and low interaction depth. Based on these parameters, the system retrieves micro-narrative units that match the optimal narrative strategy vector from the micro-narrative unit library. For example, a short text introducing the historical background of the scenic area may be accompanied by a panoramic view of the scenic area. The system connects these micro-narrative units into coherent service content and presents it to tourists in a concise text and image format through a smart guide screen, accompanied by soothing background music to reduce tourists' cognitive load.

[0069] Scene 2: Tourists visiting the scenic area When a tourist visits a certain attraction, the camera captures that the tourist's head stays on the ground for a relatively long time and the frequency of gaze drift is low. The system calculates the visual posture cognitive load index to be 0.5, which does not reach the first threshold of 0.7. Therefore, the second modality perception is not activated. Based on the visual posture cognitive load index, the system determines that the tourist is currently in the flow zone and the cognitive load state is relatively ideal. In order to maintain this state, the system maintains the current narrative strategy and periodically fine-tunes the information density parameter to ensure that the tourist's cognitive load state remains in the flow zone.

[0070] Scene 3: Tourists resting in the scenic area When tourists rest in the rest area, the camera captures that their heads remain fixed for a relatively long time and their gaze drifts less frequently. The system calculates the visual posture cognitive load index to be 0.3, which does not reach the first threshold of 0.7. Therefore, the second modality perception is not activated. Based on the visual posture cognitive load index, the system determines that the tourist is currently in an idle range with a low cognitive load. In order to improve the tourist's cognitive load to a reasonable level, the system increases the interaction depth parameter and pushes challenging question-based service content, such as fun quizzes about the history and culture of the scenic area, to stimulate the tourist's interest.

[0071] Through the above embodiments, this system can accurately perceive the cognitive load of tourists based on multimodal emotion recognition, dynamically generate personalized service content, and effectively improve tourists' experience satisfaction in cultural and tourism scenarios.

[0072] The same or similar labels correspond to the same or similar parts; The terms used to describe positional relationships in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent. Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. For those skilled in the art, other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all implementation methods here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A smart customer service response method for cultural tourism based on multimodal emotion recognition, characterized in that, Applied to a smart cultural tourism service system, the method includes: When the first modality cognitive load index reaches a first threshold, and the matching degree between the load interval to which the first modality cognitive load index belongs and the preset flow interval is less than a preset matching threshold, the second modality cognitive load sensing is activated. Based on the first modal cognitive load index and the second modal cognitive load index, the user's overall cognitive load status and load change trend are determined; When the overall cognitive load is in the overload range, the dynamic mapping algorithm based on fuzzy control logic is invoked to generate the optimal narrative strategy vector. Based on the optimal narrative strategy vector, matching micro-narrative units are retrieved from the micro-narrative unit library and concatenated to generate personalized service content, which is then presented to the user through multi-channel adaptive methods. The optimal narrative strategy vector includes parameters such as information density, presentation mode, emotional tone, and interaction depth, and the micro-narrative unit library contains structured cultural and tourism knowledge units with pre-labeled narrative parameter tags.

2. The method according to claim 1, characterized in that, The method further includes: when the matching degree between the load interval corresponding to the first modality cognitive load index and the preset flow interval is not less than the preset matching threshold, maintaining the current narrative strategy and pushing optional in-depth expansion service content.

3. The method according to claim 2, characterized in that, Before invoking the dynamic mapping algorithm based on fuzzy control logic to generate the optimal narrative strategy vector when the overall cognitive load state is in the overload range, the method further includes: determining the load range to which the overall cognitive load state belongs; When the overall cognitive load is in the flow interval, maintain the current narrative strategy vector and periodically fine-tune the information density parameter; when it is in the idle interval, increase the interaction depth parameter and push challenging question-type service content.

4. The method according to claim 1, characterized in that, The determination of the user's overall cognitive load status and load change trend includes: weighting the first modality cognitive load index and the second modality cognitive load index, wherein the weight of the behavioral interaction modality is higher than that of the visual posture modality, and the weight of the visual posture modality is higher than that of the environmental soundprint modality. A comprehensive cognitive load index is generated based on the weighted fusion results, and the load change slope of several consecutive sampling points is calculated through time series analysis to determine the load change trend.

5. The method according to claim 1, characterized in that, The method further includes: real-time monitoring of changes in the cognitive load index during the presentation of personalized service content; When the load change trend shows that the rate of decline exceeds the preset threshold, the information density parameter is gradually reduced back to the baseline value.

6. The method according to claim 1, characterized in that, The first modality includes a visual pose modality, and the second modality includes a behavioral interaction modality and an environmental voiceprint modality; The first modal cognitive load index is generated by calculating head dwell time and gaze drift frequency, while the second modal cognitive load index is generated by fusing device operation behavior characteristics and environmental sound entropy values.

7. The method according to claim 1, characterized in that, The generation of personalized service content includes: retrieving the combination of micro-narrative units with the highest matching degree from the micro-narrative unit library based on the parameter tags in the optimal narrative strategy vector; By using retrieval-enhanced generation technology, the micro-narrative units are linked into coherent content using transition language, wherein the style of the transition language is consistent with the emotional tone parameters.

8. The method according to claim 5, characterized in that, The method also includes: periodically collecting narrative strategy adjustment data and corresponding load change curves under different load ranges, and optimizing the fuzzy rule base in the dynamic mapping algorithm through cluster analysis.

9. The method according to claim 1, characterized in that, The method further includes: when the user input contains keywords for emergency help, skipping the cognitive load judgment process, directly switching to the highest priority human customer service channel, and simultaneously pushing emergency guidance content for the current scenario.

10. A smart customer service response system for cultural tourism based on multimodal emotion recognition, used to implement the method according to any one of claims 1-9, characterized in that, The system includes: The modal perception module, when the first modal cognitive load index reaches a first threshold, activates the second modal cognitive load perception when the matching degree between the load interval corresponding to the first modal cognitive load index and the preset flow interval is less than a preset matching threshold. The load analysis module determines the user's overall cognitive load status and load change trend based on the first modal cognitive load index and the second modal cognitive load index. The content generation module, when the overall cognitive load is in the overload range, calls a dynamic mapping algorithm based on fuzzy control logic to generate an optimal narrative strategy vector. Based on the optimal narrative strategy vector, it retrieves matching micro-narrative units from the micro-narrative unit library and concatenates them to generate personalized service content, which is then presented to the user through multi-channel adaptive processing. The optimal narrative strategy vector includes parameters such as information density, presentation mode, emotional tone, and interaction depth, and the micro-narrative unit library contains structured cultural and tourism knowledge units with pre-labeled narrative parameter tags.

Images