Tourist emotion real-time perception and intervention method based on multi-modal large model

Abstract

The invention discloses a tourist emotion real-time perception and intervention method based on a multi-modal large model, and particularly relates to the technical field of computer perception. The method comprises the following steps: acquiring a facial micro-expression image sequence, a gait posture sequence, a phrase-level voice stream and a body surface infrared heat map of a tourist at a scenic spot sightseeing path node, constructing a current state feature vector set F, calculating an emotion weight coefficient based on a cross-modal attention mechanism, and generating an emotion fusion representation vector E; judging the current emotional risk level R of the tourist according to E, predicting an emotional fluctuation trend delta E in a future time period based on a Shenchang differential equation model in combination with a historical path trajectory and an environment state parameter, and judging whether a critical emotional transition risk exists or not; if the risk exists, an adaptive intervention strategy is matched and executed in real time; updating the emotional state vector E'again according to the tourist intervention response data; according to the invention, high-precision and dynamic tourist emotion management can be realized, and the intelligent level of scenic spot service and the tourist safety guarantee capability are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of computer perception technology, specifically to a method for real-time perception and intervention of tourist emotions based on a multimodal large model. Background Technology

[0002] In current smart tourism management systems, tourist satisfaction assessment and service optimization mainly rely on explicit indicators, such as consumption records, check-in behavior, and online reviews. However, in high-density crowd scenarios (such as popular tourist attractions during holidays), tourists may experience drastic emotional fluctuations (such as anxiety, depression, and disappointment) in a short period of time due to factors such as long queues, sudden changes in ambient temperature, navigation difficulties, or device disconnection, but these emotions may not be identified and addressed in a timely manner through conventional means.

[0003] More seriously, some tourists are in a "subcritical state," meaning they are on the verge of a psychological breakdown even before exhibiting drastic behavior (such as complaints or conflicts). Once this emotional tipping point is triggered, it could lead to high-risk incidents such as safety accidents, mass panic, or unauthorized trespassing, becoming a highly hidden and unpredictable blind spot in the operation of smart scenic areas.

[0004] Currently, there is a lack of a systematic approach that can integrate and understand multimodal data such as tourists' micro-expressions, gait changes, voice fluctuations, and body surface heat sensations, while maintaining privacy and security, and achieving real-time emotion perception and multi-level intervention linkage. Summary of the Invention

[0005] The purpose of this invention is to provide a method for real-time perception and intervention of tourist emotions based on a multimodal large model, so as to overcome the shortcomings of the prior art.

[0006] To achieve the above objectives, the present invention provides the following technical solution: a method for real-time perception and intervention of tourist emotions based on a multimodal large model, comprising: S100: Collect multi-source modal data of target tourists at nodes along the tour route, and establish a feature vector set F of the tourist's current state; S200: Input the feature vector set F into the pre-trained cross-modal model, calculate the emotion weight coefficient αi of each modality at the current time based on the cross-modal attention mechanism, and construct the emotion fusion representation vector E, E=M(F,αi) accordingly. S300: Match the emotion fusion representation vector E with the high-risk emotion states labeled in the training set, and output the current emotion risk level R; S400, if R≥2, then based on the tourist's historical path trajectory L and current environmental state parameter W, predict the emotional fluctuation trend ΔE in the future Δt period, and determine whether there is a risk of critical emotional transition. S500, if there is a risk of critical emotional transition, call the tourist emotion intervention strategy library C, and select a matching multi-level intervention strategy P∈C based on the current R and ΔE; In S600, the updated emotion fusion representation vector E′ is recalculated based on the tourists' response data to the intervention strategy P. If E′ decreases, it indicates that the emotion has been successfully alleviated, and the effectiveness of the intervention is recorded. Otherwise, it proceeds to S200 for reassessment, forming a closed-loop adaptive adjustment mechanism.

[0007] Preferably, in S100, the tourist's current state feature vector set F includes a facial micro-expression image sequence V, a non-contact gait posture sequence G, a phrase-level speech stream S, and a body surface infrared thermogram T.

[0008] Preferably, in step S200, the cross-modal attention mechanism is used to calculate the emotion weight coefficient αi, which includes: mapping each modal feature vector to a query vector, a key vector, and a numerical vector, respectively; calculating the similarity score based on the dot product attention between modalities; and normalizing the score using the softmax function as the emotion weight coefficient.

[0009] Preferably, in step S300, an emotion level classification neural network based on a multilayer perceptron is constructed, trained using a cross-entropy loss function, and outputs probability values ​​for four categories based on the input of the emotion fusion representation vector E. Finally, the current emotion risk level R is determined based on the label corresponding to the highest probability.

[0010] Preferably, R=0 indicates emotional stability; R=1 indicates mild emotional fluctuation; R=2 indicates moderate risk requiring early warning; and R=3 indicates a high-risk state requiring immediate intervention.

[0011] Preferably, in step S400, an emotion evolution function is constructed based on Neural Ordinary Differential Equations, the derivative of the emotion state with respect to time is defined as a trainable neural network function, and the Runge-Kutta method is used to perform numerical integral prediction within the future time period Δt to obtain the future emotion state vector and calculate the emotion fluctuation trend ΔE.

[0012] Preferably, the inputs to the Neural Ordinary Differential Equations include the current emotional state E, the path trajectory vector L_embed, and the standardized environmental state parameters W. The magnitude of the change in the emotional state is obtained by calculating the Euclidean distance between the current emotional state E and the predicted state E(t+Δt).

[0013] Preferably, the intervention strategies in the tourist emotion intervention strategy library C are matched and screened according to the emotion risk level R and the emotion fluctuation trend ΔE, and sorted according to the priority of historical intervention feedback effects.

[0014] Preferably, in step S600, the updated emotion fusion representation vector E′ is calculated by re-collecting the multimodal data after the intervention, and the success of the intervention is determined by whether the change in vector magnitude exceeds the relief threshold θ_relief. If it is unsuccessful, the process automatically returns to step S200 to re-execute the emotion perception process.

[0015] The technical effects and advantages provided by the present invention in the above technical solution are as follows: 1. This invention, by introducing a multimodal data perception and fusion modeling mechanism, can comprehensively acquire multi-source emotion-related information such as facial expressions, gait posture, voice features, and body surface temperature of tourists at different nodes along the tour route, effectively overcoming the limitations of traditional single-modal monitoring in terms of accuracy and robustness. By constructing a multimodal large-scale model with a cross-modal attention mechanism, it can dynamically allocate weights according to the actual contribution of different modalities to the current emotional state, generating a highly discriminative emotion fusion representation vector, thereby significantly improving the real-time performance and accuracy of tourist emotion recognition, and providing a reliable basis for subsequent graded response and individualized intervention.

[0016] 2. This invention introduces an emotion trend prediction model based on neuronormal differential equations. By combining tourists' historical path trajectories with current environmental state parameters, it can proactively detect and dynamically intervene before emotional shifts occur, effectively reducing potential safety or service risks caused by emotional outbursts. Simultaneously, the system achieves adaptive evaluation and continuous optimization of intervention strategies through a data-driven closed-loop update mechanism, demonstrating good generalization ability and scalability. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0018] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] For examples, please refer to Figure 1 As shown in this embodiment, the real-time tourist emotion perception and intervention method based on a multimodal large model includes: S100: Collect multi-source modal data of target tourists at nodes along the tour route and establish a feature vector set F of the tourist's current state.

[0021] In this embodiment, the real-time tourist emotion perception and intervention method based on a multimodal large model is collaboratively completed by multimodal acquisition terminals deployed at the scenic area entrance, main tour routes, and around key attractions, and by the scenic area edge computing server. Step S100 specifically includes the following: Several tour path nodes P1, P2, ..., Pn are pre-divided on the main tour route of the scenic area. Each tour path node corresponds to a space area where tourists are more likely to stay or pass through, such as: ticket gate, viewing platform, queuing area, intersection, and service point.

[0022] At each node along the tour route, at least one multimodal data acquisition terminal is installed, the multimodal data acquisition terminal comprising: Visible light camera unit: used to capture facial images and full-body posture images of tourists; Infrared thermal imaging unit: used to acquire infrared thermal images of tourists' bodies; Environmental sound pickup unit: used to collect sound signals containing tourists' phrase-level speech; Local processing and caching unit: used to synchronize, trim, and anonymize the above raw data, and then package and upload the processed data to the scenic area edge server.

[0023] To facilitate subsequent tracking of emotional states, each tourist obtains an anonymous tourist ID decoupled from their real identity through a ticket purchase QR code or electronic wristband upon entering the scenic area. When the multimodal data collection terminal at the tour route node captures tourist data, it binds the anonymous tourist ID to the current collection timestamp.

[0024] When a target tourist passes through any node on the tour route, the visible light camera unit automatically locks the tourist's facial region based on a face detection and tracking algorithm, and continuously acquires multiple frames of facial images within a preset time window Δt1 (e.g., 1 to 3 seconds) to form a facial image sequence V_raw.

[0025] Preferably, in this embodiment, the facial image sequence V_raw is preprocessed as follows: A facial landmark localization algorithm is used to align and normalize each frame of the image, removing frames with excessive pose deviations. The aligned facial images are then scaled to a uniform resolution, such as 112×112 pixels, to reduce computational overhead. Micro-expression magnification algorithms (such as temporal motion magnification) are used to salience subtle muscle changes in areas such as the corners of the eyes and mouth. The processed N consecutive frames are arranged chronologically, denoted as V = {v1, v2, ..., vN}, forming a facial micro-expression image sequence V. The sequence length N can be adaptively adjusted according to the node dwell time, and is preferably set within the range of 8 to 32 frames, ensuring that at least one complete micro-expression change cycle is captured.

[0026] At each node along the same tour route, a visible light camera unit or a separately configured wide-angle camera unit is used to capture the walking process of tourists from the node entrance to the exit. Using human detection and multi-object tracking algorithms, consecutive walking image frames matching anonymous tourist IDs are extracted to form a walking image sequence G_raw.

[0027] To obtain more stable gait posture information, this embodiment performs the following processing on the walking image sequence G_raw: Using a two-dimensional or three-dimensional human skeleton key point recognition model, the coordinates of the main joints in each frame, such as the head, torso, upper and lower limbs, are extracted to obtain the skeleton pose vector p_t. The attitude vector sequence is interpolated and resampled at a fixed time interval Δt2 (e.g., 0.1 seconds) to eliminate the impact of frame rate fluctuations. The gait posture sequence G = {p1, p2, ..., pM} is obtained by concatenating the posture vectors of M consecutive time steps in chronological order. The value of M can be automatically adjusted according to the actual walking distance of the tourist at the node. Preferably, when abnormal gait characteristics such as decreased speed, lingering in place, or sudden stopping are detected in the tourist within the node, the sequence length is automatically extended to more comprehensively characterize emotion-related posture changes.

[0028] The environmental sound pickup unit continuously collects environmental sound signals at a preset sampling rate (e.g., 16kHz) within the tour path nodes and runs speaker separation and speech activity detection algorithms in real time. When a speech event corresponding to the spatial location of the target tourist is detected, the tourist's speech segment is separated from the environmental sound source to form the raw speech stream S_raw.

[0029] To meet the dual requirements of privacy and emotion recognition, this embodiment restricts the voice content to the "phrase level" and performs the following processing: S_raw is segmented into segments with a time window Δt3 (e.g., 0.5–2 seconds), each segment corresponding to a speech phrase or intonation change unit. Noise reduction and endpoint detection are performed on each speech phrase to remove background noise and silent segments. Emotion-related acoustic features such as Mel-frequency cepstral coefficients (MFCC), fundamental frequency profile, energy changes, and pitch fluctuations are calculated based on acoustic feature extraction algorithms, and the corresponding phrase-level speech waveforms are preserved. The preprocessed K consecutive phrases are organized in chronological order into S = {s1, s2, …, sK} to form a phrase-level speech stream S, without preserving semantic text content to reduce the impact on visitor privacy.

[0030] When a tourist enters the effective monitoring area of ​​a node on the tour route, the infrared thermal imaging unit automatically triggers the acquisition of an infrared thermal image of the tourist's upper body. To reduce the interference of ambient temperature fluctuations on emotion recognition, this embodiment pre-calibrates the baseline ambient temperature at each node and performs relative temperature normalization processing on the acquired thermal images. Specifically, this includes: Human body region segmentation is performed on the original infrared image to remove the background region; Local thermal patching of temperature-sensitive areas such as the forehead, eye sockets, nose wings, neck, and hands is extracted from the segmented human body region; The above local heatmaps are interpolated and smoothed to generate a temperature distribution matrix of uniform size; Several frames of thermal images collected within the same time window are superimposed or compressed in chronological order to obtain the body surface infrared thermal image T.

[0031] In a preferred embodiment, if the visitor stays at a node for a short time, only one key heat map frame can be retained as T; if the stay is long, a time series consisting of multiple heat maps is generated to improve the sensitivity to subtle changes in body surface temperature related to emotions.

[0032] After acquiring the facial micro-expression image sequence V, non-contact gait posture sequence G, phrase-level speech stream S, and body surface infrared thermogram T, this embodiment aligns and synchronizes the four modal data based on a unified timestamp, marks the missing modalities, and performs masking processing.

[0033] For data belonging to the same anonymous tourist ID within the same tour route node and the same time window, a current state feature vector set is formed for that node: F={V,G,S,T}; where: V is used to represent the instantaneous change pattern of the tourist's facial micro-expressions; G is used to represent the dynamic characteristics of the tourist's overall gait and posture; S is used to represent the emotional tone of the tourist's voice when communicating or talking to themselves; and T is used to represent the distribution of the tourist's body surface temperature and its subtle fluctuations related to emotions.

[0034] Finally, the completed feature vector set F, along with the tourist's anonymous tourist ID and the current tour path node number, is packaged and uploaded to the scenic area's edge computing server. This serves as the basic data for inputting the multimodal large model in subsequent steps, used to calculate the emotion fusion representation vector E and further conduct emotion risk assessment and intervention strategy generation.

[0035] S200: Input the feature vector set F into the pre-trained cross-modal model, calculate the emotion weight coefficient αi of each modality at the current time based on the cross-modal attention mechanism, and construct the emotion fusion representation vector E, E=M(F,αi).

[0036] After obtaining the tourist's current state feature vector set F={V,G,S,T}, this embodiment further constructs a cross-modal emotion understanding network with context modeling capabilities, dynamically assigns weights to emotion information from different modalities, and merges them into a unified emotion representation vector E, providing accurate input for subsequent emotion level assessment and intervention strategies.

[0037] The steps consist of four consecutive sub-steps: modal embedding vector extraction, cross-modal correlation modeling, weight normalization processing, and sentiment fusion representation generation, as detailed below: To accommodate the differences in data structures between modalities, four modal feature encoding structures were designed for images, sequences, audio, and heatmaps. The encoders for each modality are described in detail below: The input consists of n consecutive frames of face-aligned images, all with a uniform size of 112×112 pixels. It employs a 3D convolutional network structure based on spatiotemporal feature extraction, including: First layer: 3D convolution kernel size is (3×3×3), number of channels is 64; Second layer: 3D convolution kernel size is (3×3×3), number of channels is 128; Following this is a time-dimensional pooling and fully connected layer, the output dimension is 128; The network is pre-trained using publicly available facial micro-expression recognition datasets (such as CASME II and SAMM) and fine-tuned on real-world tourist facial images.

[0038] The output is a facial modality embedding vector V_embed, which has 128 dimensions.

[0039] The key points of the human skeleton in each frame of the gait posture sequence are constructed as a graph structure, where nodes represent joints and edges represent joint connections. Input a sequence of graph structures into a two-layer graph convolutional network, and set the convolutional kernel to a normalized form of adjacency matrix plus identity matrix; Average pooling is performed on the graph embedding along the time dimension to obtain the gait mode embedding vector G_embed, which has a dimension of 128.

[0040] Extract audio frame-level features such as Mel-frequency cepstral coefficients (MFCC), fundamental frequency, and energy variation from each frame of a speech segment; Frame-level features are input into a temporal modeling structure consisting of a two-layer bidirectional long short-term memory network (Bi-LSTM), with a unidirectional dimension of 64 in the hidden layer; At the output position of the last frame, a fully connected layer is added to output a speech embedding vector S_embed with a dimension of 128.

[0041] Input an infrared thermal image, 64×64 pixels in size, and use a 2-layer 2D convolutional neural network: The first convolutional kernel has a size of (5×5) and 32 channels; The second convolutional kernel has a size of (3×3) and 64 channels; The fully connected layer outputs a heatmap modal embedding vector T_embed, which has 128 dimensions.

[0042] Ultimately, a unified set of modal embeddings is formed: F_embed={V_embed,G_embed,S_embed,T_embed}.

[0043] After obtaining all modal embedding vectors, they are input into a cross-modal attention modeling structure to calculate the relative importance of each modality to the current emotional state. This structure includes the following steps: For each modality embedding vector X∈F_embed, three linear transformation layers are used to obtain: query vector Qx=W_Q·X; key vector Kx=W_K·X; value vector Vx=W_V·X; where W_Q, W_K, and W_V are trainable weight matrices with dimensions (128×64).

[0044] For mode i and mode j, the attention score is defined as: Where “·” represents the dot product operation, d is the vector dimension of 64, and softmax normalizes across all modalities. The score matrix A is 4×4 in dimension, Kj represents the key vector of the j-th modality (such as speech, image, etc.); T is the vector transpose.

[0045] For each mode i, the fused output is calculated as follows: The formation reflects the results of its cross-modal information integration.

[0046] A fully connected layer is used for the output vector Oi of each modality to output a scalar ri∈[0,1], which represents the relevance of the modality to the current emotional state.

[0047] The initial set of emotion relevance scores is finally obtained: r={r1,r2,r3,r4}.

[0048] The cross-modal model employs a self-supervised pre-training approach during the training phase. First, the modal encoder is pre-trained based on a publicly available emotion database. Then, the attention fusion module and emotion classifier are fine-tuned under supervision using multimodal samples collected from actual scenic areas. The training sample size is approximately 10,000 sets, each containing synchronously collected image, speech, posture, and infrared heatmap data, labeled with emotion level tags by psychology experts.

[0049] To enhance the contextual coherence and decision robustness of sentiment relevance scores, we further incorporate information about the preceding and following nodes within the time window, as well as sentiment supervision signals, to optimize modality weights. Take the historical representation of the tourist's emotion vectors from the two consecutive nodes before the current node, denoted as E(t-1) and E(t-2). Input E(t-1), E(t-2) and the current r together into a bidirectional gated recurrent unit network to extract time series features T_seq with a dimension of 64.

[0050] Concatenate T_seq and r, input to a fully connected layer, and output vector A_raw={a1,a2,a3,a4} as unnormalized sentiment weights.

[0051] The Softmax function is used to normalize A_raw, resulting in the final set of sentiment weight coefficients: , where i,j∈{1,2,3,4}; Output: α1 (face), α2 (gait), α3 (speech), α4 (infrared).

[0052] The modal embedding vector F_embed is weighted and fused with the set of weight coefficients {α1, α2, α3, α4} to construct a unified sentiment representation vector E. The specific method is as follows: ; where multiplication is the product of a scalar and a vector, and the final dimension of E is 128, which serves as a unified vector representation of the current tourist's emotional state.

[0053] This vector not only integrates multimodal perceptual features but also, guided by context and supervision signals, possesses stronger semantic representation and emotion discrimination capabilities. Vector E will serve as the core input data for the emotion risk prediction and intervention decision-making module.

[0054] S300 matches the emotion fusion representation vector E with the high-risk emotion states labeled in the training set and outputs the current emotion risk level R.

[0055] After constructing the emotion fusion representation vector E, to achieve automated identification and hierarchical classification of tourist emotional risk, this embodiment matches E with a pre-constructed emotion labeling training set through an emotion level classification network and outputs the tourist's current emotional risk level R. This process includes four consecutive steps, as detailed below: Obtain a set of historical tourist emotion sample data with expert-annotated labels to supervise the training of the emotion level classification model.

[0056] Sample source: Collect feature vectors F={V,G,S,T} of different tourists at multiple path nodes in real scenic spots, and construct the emotion fusion representation vector E through the aforementioned steps to obtain the emotion vector sample set.

[0057] Manual annotation method: A team of annotation experts with backgrounds in psychology and behavioral analysis will classify the emotional state of each sample by combining video observation, audio content, tourist behavior results (such as complaints, leaving the group, conflicts, etc.) and self-reported feedback.

[0058] Labeling rules: Set 4 levels of emotional state labels, namely: Level 0: Stable mood, no obvious fluctuations; Level 1: Mild discomfort, with slight irritability or fatigue; Level 2: Obvious anxiety, with rapid speech, abnormal gait, etc.; Level 3: On the verge of collapse, with obvious risk of emotional loss of control (such as staying still, using strong language, or leaving the crowd).

[0059] Each sample data point includes an emotion fusion representation vector E (128 dimensions) and a corresponding emotion level label L∈{0,1,2,3}, forming a training set D={(E1,L1),(E2,L2),...,(En,Ln)}. This training set is used to guide subsequent classification models in learning the mapping relationship between multimodal emotion vectors and risk levels.

[0060] Design a neural network for emotion classification with nonlinear fitting capabilities to predict the risk level of an input emotion fusion representation vector E.

[0061] Network Input: The input layer receives a sentiment fusion representation vector E with a dimension of 128. Network Structure: The emotion classification network uses a three-layer fully connected neural network structure, as follows: First hidden layer: 128 neurons, activation function is rectified linear unit function; The second hidden layer consists of 64 neurons, with the rectified linear unit function as the activation function. Output layer: 4 neurons, corresponding to 4 emotion level labels, the output is a 4-dimensional real vector P={p0,p1,p2,p3}, where pi represents the probability prediction value of belonging to level i, i=0,1,2,3.

[0062] Training method: Supervised training is performed using training set D, and the loss function is the cross-entropy loss function. The model parameters are optimized by backpropagation and gradient descent until convergence.

[0063] Training parameter settings: learning rate set to 0.001; batch size to 64; number of training epochs to 100; adaptive moment estimation algorithm selected as the optimizer. Through this structure, the model possesses the ability to perform nonlinear mapping and classification of multimodal fusion features.

[0064] Once training is complete, the model can infer the emotion fusion representation vector E obtained in real time and output the emotion level.

[0065] Forward reasoning process: Input the current tourist's emotion fusion representation vector E into the emotion classification network to obtain the output probability vector P={p0,p1,p2,p3}; Maximum probability rule: Traverse the four elements of vector P and select the index corresponding to the maximum value as the predicted sentiment level label L_pred. That is, if pk=max(p0,p1,p2,p3), then L_pred=k. Here, k is an integer between 0 and 3, representing the sentiment level to which the prediction result belongs.

[0066] The aforementioned emotion level label L_pred is used as the current tourist's emotion risk level R. The value of R ranges from 0 to 3, with the following specific meanings: R=0 indicates stable emotion; R=1 indicates mild emotion fluctuation; R=2 indicates moderate risk requiring warning; R=3 indicates a high-risk state requiring immediate intervention. The emotion risk level R will serve as the core decision parameter for emotion trend prediction and intervention strategy selection in subsequent steps.

[0067] This embodiment constructs a generalizable and effective method for judging emotional risk levels by supervising the mapping relationship between the emotion fusion representation vector E and the expert-annotated training samples. It has the technical advantages of clear model structure, clear label logic, and stable classification method, and can achieve accurate, real-time, and graded emotional risk identification in the dynamic emotional perception of tourists.

[0068] S400, if R≥2, then based on the tourist's historical path trajectory L and current environmental state parameter W, predict the emotional fluctuation trend ΔE in the future Δt period, and determine whether there is a risk of critical emotional transition.

[0069] Before performing sentiment trend prediction, initial conditions for sentiment states are first constructed for continuous-time modeling.

[0070] The emotion fusion representation vector Et calculated at the current tour path node is used as the initial emotion state vector. This emotion fusion representation vector is a fixed-dimensional vector with a dimension of 128, which is used to represent the tourist's comprehensive emotion state at the current moment. Extract the sequence information of the K most recent tour path nodes from the tourist's historical path trajectory L, where K ranges from 3 to 5. The path trajectory information includes node number, node stay duration, and node visit order. The above path trajectory information is mapped to a path feature vector L_embed through position embedding. The path feature vector and the emotion fusion representation vector are concatenated in dimension to form an extended emotion state vector E0. The current environmental state parameters W are standardized. These environmental state parameters include ambient temperature, relative humidity, ambient noise intensity, and population density per unit area. The standardized environmental state parameters are used as external input variables to influence the emotional evolution process.

[0071] Thus, the initial emotional state E0 and the corresponding environmental state parameter W for continuous time prediction are obtained.

[0072] In this step, a Neural Ordinary Differential Equations model is constructed to describe the continuous changes in emotional states over time.

[0073] The process of emotional state changing over time is defined as a first-order ordinary differential equation, where the rate of change of emotional state with respect to time is represented by a learnable nonlinear function. The nonlinear function is implemented by a multi-layer fully connected neural network, which includes: The input layer takes as input the concatenation of the current emotion state vector and the environmental state parameter vector. The first hidden layer has 128 neurons, and the activation function is the rectified linear unit function. The second hidden layer has 128 neurons. The output layer outputs the same dimension as the emotion state vector, used to represent the change in emotion state per unit time. The parameters of this neural network are trained using historical emotion trajectory samples, with the training objective being to minimize the error between the model's predicted emotion evolution and the actual emotion evolution trajectory. This neural network is embedded as an emotion evolution function into an ordinary differential equation to form a complete Neural Ordinary Differential Equations emotion prediction model.

[0074] The above model is used to describe the changing pattern of emotional state over a continuous time axis, given the current emotional state and environmental conditions.

[0075] After the model is built, numerical predictions are made for the changes in sentiment over the future time range of Δt.

[0076] Set the prediction time window Δt, and its value is set to 1 to 5 minutes according to the scenic area management needs; Using the initial emotional state E0 obtained in step one as the starting condition, and the environmental state parameter W as a constant external input; The emotion evolution equation is integrally calculated using an explicit ordinary differential equation numerical solution method, specifically the Runge–Kutta method with an adaptive step size. During the integration process, the trajectory of the change of emotional state on the time axis is continuously calculated to obtain the emotional state vector E(t+Δt) corresponding to the predicted time t+Δt. The predicted emotional state vector E(t+Δt) is compared with the current emotional fusion representation vector Et. The difference is the emotional fluctuation trend ΔE, which is used to characterize the magnitude and direction of emotional changes in the future time period Δt.

[0077] After obtaining the emotional fluctuation trend ΔE, it is determined whether there is a risk of a critical emotional transition.

[0078] The magnitude index of change is calculated for the emotional fluctuation trend ΔE. The magnitude index is defined as the magnitude of the ΔE vector and is used to quantify the intensity of emotional changes. The critical transition threshold θ is obtained in advance based on the statistical analysis of historical emotion evolution samples. The critical transition threshold is the upper limit of the average change range corresponding to the transition of emotion level from 2 to 3 in the training samples. Compare the currently predicted magnitude of change with the critical transition threshold θ: When the magnitude of change is greater than or equal to the critical threshold, it is determined that there is a risk of a critical emotional transition. When the magnitude of change is less than the critical threshold, it is determined that there is no risk of a critical emotional transition. The judgment result will be used as the input parameter for triggering subsequent emotion intervention strategies.

[0079] The critical transition threshold θ is determined through statistical analysis of the change in the magnitude of the E vector in the emotion level transition (level 2 → level 3) samples in the training set, with the 85th percentile selected as the upper limit of the threshold. The relief threshold θ_relief is set by the median decrease in the E vector in the intervention samples, and supports adaptive adjustment of ±10% based on the user ID's historical behavior.

[0080] S500, if there is a risk of critical emotional transition, then call the tourist emotion intervention strategy library C, and select a matching multi-level intervention strategy P∈C based on the current R and ΔE.

[0081] Having predicted the emotional fluctuation trend ΔE and determined the existence of a critical emotional transition risk, this embodiment further executes a targeted intervention process to prevent the emotional state from transitioning from a high-risk level to the edge of being out of control. This process involves calling a pre-set tourist emotion intervention strategy library C, and selecting an appropriate multi-level emotion intervention strategy P based on the current emotional risk level R and the emotional fluctuation trend ΔE. This strategy is used to provide non-contact, non-invasive proactive emotional guidance and risk mitigation for tourists.

[0082] The emotion intervention strategy library C consists of several predefined emotion intervention strategies. Each strategy Pi includes: the applicable range of emotional risk levels R_range; the tolerable range of emotional fluctuation intensity ΔE_range; intervention triggering conditions; definition of intervention execution actions; and intervention feedback collection methods. Emotion intervention strategies are divided into three levels according to intervention intensity and intervention method: Level 1 strategy: Mild intervention, such as pushing soothing music, static meditation images, or whispered reminders to the visitor's mobile device interface; Secondary strategy: Moderate intervention, such as recommending alternative routes to avoid crowded areas and sending "low-density area guidance maps"; Level 3 strategy: Advanced intervention, such as dispatching inspection personnel to conduct non-contact approach inspections, or activating environmental adjustment devices (such as ventilation, background music adjustment). Each strategy Pi has a strategy ID, which is used for indexing and scheduling execution.

[0083] The calculated emotional risk level R is used as the first-level screening condition for intervention triggering; the strategy library C is traversed to extract the set C_R of all strategies that satisfy R∈R_range; if R=2 (moderate risk), the first-level and second-level strategies are selected first; if R=3 (high risk), the third-level strategy candidates are enabled at the same time; the screening result C_R is used as the candidate set for the next step of strategy fine matching.

[0084] For each strategy Pi in the selected candidate strategy set C_R, read its ΔE_range parameter; calculate the amplitude of the current sentiment fluctuation trend ΔE, i.e., the Euclidean modulus of ΔE; retain only strategies that satisfy ΔE∈ΔE_range to form the final feasible strategy set C_final; if C_final contains multiple strategies, sort and select them according to the following priority: Preferential strategies with a high historical positive feedback rate (based on intervention effectiveness records); If the intervention effects are similar, the low-intrusion strategy that causes the least disturbance to tourists should be given priority. If multiple strategies are listed, randomly select one as the current strategy P.

[0085] The intervention parameters in the selected strategy P are sent to the visitor terminal or the equipment at the edge of the scenic area. Perform specific intervention operations according to the action types defined in P, such as: reminding tourists to rest by vibrating the wearable terminal they wear; sending stress-relieving pictures or navigation suggestions to tourists' WeChat mini programs; adjusting the rhythm and brightness of the background music in the area where tourists are located; arranging staff to patrol and observe tourists' reactions; if strategy P contains a feedback collection mechanism, then collect data on changes in tourists' status after the intervention, including but not limited to changes in facial expressions, gait adjustments, voice tone stability, and position changes, as the basis for evaluating the subsequent intervention loop.

[0086] In S600, the updated emotion fusion representation vector E′ is recalculated based on the tourists' response data to the intervention strategy P. If E′ decreases, it indicates that the emotion has been successfully alleviated, and the effectiveness of the intervention is recorded. Otherwise, it proceeds to S200 for reassessment, forming a closed-loop adaptive adjustment mechanism.

[0087] After implementing the multi-level emotion intervention strategy P, to evaluate the intervention effect and continuously optimize the intervention strategy, this embodiment proposes an emotion state update method based on tourist response data. A closed-loop adaptive adjustment mechanism is constructed based on this update result to achieve dynamic tracking and multi-round intervention evaluation of tourist emotion states. Specifically, the following steps are included: Within a preset observation period T_obs (T_obs is generally set to 30-90 seconds) after the execution of the emotion intervention strategy P, the multimodal sensing device deployed at the current tour node collects the tourists' behavior and physiological state data again, forming the tourist response dataset R_data. The response modality content includes: facial micro-expression image sequence; gait posture behavior sequence; voice segments; infrared heat map; spatial behavior data (such as whether the route is changed, changes in stay duration); the collection of the response data is consistent with the collection method of the tourist state feature vector set F in step S100, ensuring the consistency of the data processing path before and after; the collected data is processed by a modality-specific encoding network to extract modality embedding features, and input into a pre-trained multimodal large model, and step S200 is repeated to calculate the emotion fusion representation vector E′ after intervention.

[0088] Let E be the emotion fusion representation vector before intervention, and E′ be the emotion fusion representation vector after intervention, with both having the same dimension; calculate the magnitude of emotion change ΔE′ between them, which is defined as the Euclidean distance between E and E′, i.e.: ;in, The L2 norm represents the Euclidean distance between vectors; to facilitate the assessment of mitigation effectiveness, a directional criterion is set for ΔE′: If ΔE′ changes negatively and the magnitude of E′ is significantly lower than that of E, it indicates that the mood tends to be stable. If ΔE′ is positive or does not decrease significantly, it indicates that the intervention has limited effect.

[0089] The effectiveness of the intervention is determined by comparing the value of ΔE′ with the preset relief threshold θ_relief.

[0090] The relief threshold θ_relief is set based on historical intervention sample experience data and is the minimum effective difference in the decrease of the emotional state modulus (e.g., set to 2.0); the judgment rule is as follows: If ΔE′≥θ_relief, the intervention is considered successful; otherwise, the intervention is considered ineffective. The judgment result, along with the corresponding intervention strategy number P, the anonymous tourist ID, the intervention timestamp, and other information, is recorded to form an intervention feedback sample, which is used for subsequent strategy library optimization and model retraining.

[0091] After completing the evaluation of the intervention's effectiveness, the system selects subsequent treatment paths based on the assessment results, thus forming a closed-loop adjustment process: If the intervention is deemed effective (E′ decreases), the emotional state shows a good recovery trend, the system suspends further intervention, and maintains the state in normal monitoring mode; If the intervention is deemed ineffective, the system automatically re-enters step S200, which involves: using E′ as the new current emotional state; recalculating the weights, determining the risk level, and predicting the emotional trend; reselecting or upgrading the intervention strategy P′∈C to form a second round of intervention cycle; and automatically accumulating the intervention history after each round of intervention to gradually form an individualized intervention response model for tourists, thereby achieving adaptive adjustment and strategy optimization.

[0092] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for real-time perception and intervention of tourist emotions based on a multimodal large model, characterized by: include: S100: Collect multi-source modal data of target tourists at nodes along the tour route, and establish a feature vector set F of the tourist's current state; S200: Input the feature vector set F into the pre-trained cross-modal model, calculate the emotion weight coefficient αi of each modality at the current time based on the cross-modal attention mechanism, and construct the emotion fusion representation vector E, E=M(F,αi) accordingly. S300: Match the emotion fusion representation vector E with the high-risk emotion states labeled in the training set, and output the current emotion risk level R; S400, if R≥2, then based on the tourist's historical path trajectory L and current environmental state parameter W, predict the emotional fluctuation trend ΔE in the future Δt period, and determine whether there is a risk of critical emotional transition. S500, if there is a risk of critical emotional transition, call the tourist emotion intervention strategy library C, and select a matching multi-level intervention strategy P∈C based on the current R and ΔE; In S600, the updated emotion fusion representation vector E′ is recalculated based on the tourists' response data to the intervention strategy P. If E′ decreases, it indicates that the emotion has been successfully alleviated, and the effectiveness of the intervention is recorded. Otherwise, it proceeds to S200 for reassessment, forming a closed-loop adaptive adjustment mechanism.

2. The method for real-time perception and intervention of tourist emotions based on a multimodal large model according to claim 1, characterized in that: In S100, the tourist's current state feature vector set F includes a facial micro-expression image sequence V, a non-contact gait posture sequence G, a phrase-level speech stream S, and a body surface infrared thermogram T.

3. The method for real-time perception and intervention of tourist emotions based on a multimodal large model according to claim 1, characterized in that: In step S200, the cross-modal attention mechanism is used to calculate the emotion weight coefficient αi, which includes: mapping each modal feature vector to a query vector, a key vector, and a numerical vector, respectively; calculating the similarity score based on the dot product attention between modalities; and using the softmax function to normalize the score and use it as the emotion weight coefficient.

4. The method for real-time perception and intervention of tourist emotions based on a multimodal large model according to claim 1, characterized in that: In step S300, an emotion level classification neural network based on a multilayer perceptron is constructed, trained using the cross-entropy loss function, and outputs the probability values ​​of four categories based on the input of the emotion fusion representation vector E. Finally, the current emotion risk level R is determined based on the label corresponding to the highest probability.

5. The method for real-time perception and intervention of tourist emotions based on a multimodal large model according to claim 4, characterized in that: wherein, R=0 indicates stable emotions; R=1 indicates mild emotional fluctuations; R=2 indicates moderate risk, requiring early warning; R=3 indicates a high-risk state, requiring immediate intervention.

6. The method for real-time perception and intervention of tourist emotions based on a multimodal large model according to claim 1, characterized in that: In S400, an emotion evolution function is constructed based on Neural Ordinary Differential Equations. The derivative of the emotion state with respect to time is defined as a trainable neural network function. The Runge-Kutta method is used to predict the numerical integral within the future time period Δt, thereby obtaining the future emotion state vector and calculating the emotion fluctuation trend ΔE.

7. The method for real-time perception and intervention of tourist emotions based on a multimodal large model according to claim 6, characterized in that: The inputs to the Neural Ordinary Differential Equations include the current emotional state E, the path trajectory vector L_embed, and the standardized environmental state parameters W. The magnitude of the change in emotional state is obtained by calculating the Euclidean distance between the current emotional state E and the predicted state E(t+Δt).

8. The method for real-time perception and intervention of tourist emotions based on a multimodal large model according to claim 1, characterized in that: The intervention strategies in the tourist emotion intervention strategy library C are matched and filtered according to the emotion risk level R and the emotion fluctuation trend ΔE, and sorted according to the priority of historical intervention feedback effects.

9. The method for real-time perception and intervention of tourist emotions based on a multimodal large model according to claim 1, characterized in that: In step S600, the updated emotion fusion representation vector E′ is calculated by re-collecting the multimodal data after the intervention, and the success of the intervention is determined by whether the change in vector magnitude exceeds the relief threshold θ_relief. If it is unsuccessful, the process automatically returns to step S200 to re-execute the emotion perception process.

Images